Download
We're evolving to serve you better! This current forum has transitioned to read-only mode. For new discussions, support, and engagement, we've moved to GitHub Discussions.

Is it there a way to selectively upload only new items?

  • #3314
    Avatar photo[anonymous]

    In the first days, when I started using Publii, I found it too fast to publish my page.

    With a few weeks using it, I have found the upload takes more and more time, as it publishes the whole website and not only the new or updated/modified posts and its contents.

    Is it there a way to just upload the affected new or updated/modified items?

    I know that if I publish a new post, the main page and other elements will be modified or affected by this new post. So it would be nice to have these elements updated at the time I upload a new post.

    But the rest of the static pages/elements will be the same. So I guess there should be a way to do what I say, maybe already solved but I haven’t noticed that options so far.

    Thank you very much.

    #3316
    Avatar photoBob

    Hi geppettvs,

    There are many elements that affect the length of the upload process, for example, Related posts – when you add a new post the list of related posts changes on all post pages.

    Have a look at the “How to speed up rendering process?” article, it may help you optimize your upload time.

    #3317
    Avatar photo[anonymous]

    Hey Bob, thank you very much for your answer.

    I understand there will be many things which will be affected by new posts. And I understand that will take a lot of time which I don’t care to spend if that’s a major change to the blog.

    But look.

    Sometimes I misspell a word, or let’s say: If I want to change a picture, removing an old picture and placing a new one.

    If I modify a single character to the newest post, having in mind that everything else will be the same. The affected post has just a character difference, but the whole page will be uploaded again.

    I guess the only way to do such thing is publishing my blog to a local folder, then manually uploading via FTP with the proper FTP client, which will ask what to do with current files on the server, so I can answer “If size differs” or not to overwrite current files on the server.

    Anyway, as Publii solves by itself the FTP task, it would be great to have this options when uploading a correction like the one I mentioned.

    Thank you!

    #3323
    Avatar photo[anonymous]

    @geppettvs – if Publii uploads many files due small change on the post there are two possibilities:

    1. Your small change affects many subpages and they should be updated anyway.
    2. There is a problem on your server with uploading files.publii.json file – this file is used to compare file contents checksums, if file not exists on the list or checksum is different – Publii uploads a file.
    #3325
    Avatar photo[anonymous]

    Hello Tomasz.

    Thank you for your answer.

    Your point 2 sounds technically logic. Nevertheless I haven’t done anything to modify the behavior of Publii on this point.

    So now I wonder if there’s something I should have done before in order to make Publii work faster.

    If so, can you please point me to an online doc where to find such information?

    Let me say that I am just working with the default Publii options. So once I finish writing my new post, or comiting my minor changes to an old post, I just press the “Sync your website” button and when I see the “Website synchronization” window I hit the “Sync your website” again.

    I have noticed there is a warning which reads:

    Remember! Any duplicate files or filenames already in the destination location that match the Publii-generated files will be overwritten.

    Which makes me think that, if -as mentioned before- I change a single character (misspelled word or something), the whole images and other stuff will be uploaded.

    This should be solved by the comparison method you mentioned in your point 2, but I don’t know how to check if it is working on my end.

    I am using Linux, Ubuntu 18.04 LTS and Publii is running pretty fine on this OS.

    Any ideas are welcome.

    Thanks in advance.

    #3338
    Avatar photo[anonymous]

    How many files is uploaded during sync? And how many files has your website at all?

    #3345
    Avatar photo[anonymous]

    Hello Tomasz.

    I am attaching a screenshot of my Publii uploading the blog. I didn’t commit any change, didn’t change anything. Just opened the application and hit the “Sync your website” button as I originally mentioned.

    uploading-with-no-mofifications

    As you can see, Publii is uploading the whole website. This time it is showing “Uploading website (nnn of 479 operations done)” but If I publish new posts the number of operations will increase.

    I mean: the whole website is being uploaded, no matter what I do.

    Thank you very much.

    #3348
    Avatar photo[anonymous]
    [anonymous] wrote:

    Hello Tomasz.

    I am attaching a screenshot of my Publii uploading the blog. I didn’t commit any change, didn’t change anything. Just opened the application and hit the “Sync your website” button as I originally mentioned.

    uploading-with-no-mofifications

    As you can see, Publii is uploading the whole website. This time it is showing “Uploading website (nnn of 479 operations done)” but If I publish new posts the number of operations will increase.

    I mean: the whole website is being uploaded, no matter what I do.

    Thank you very much.

    I am now attaching a new screenshot. Publii finished uploading the whole website 15 minutes after I posted my previous answer.

    uploading-finished

    So, I guess it is not doing the file checksum.

    Any ideas?

    #3353
    Avatar photo[anonymous]

    Could you check if there is files.publii.json file on your server and if it is not corrupted?

    #3359
    Avatar photo[anonymous]

    Tomasz,

    I have checked on my web server and I can see the file files.publii.json does exist in the root of the blog folder, as you can see highlighted on this screenshot.

    publii-the-file-exists

    The file contents seems not to be corrupted, as I can open this file in nano and the file contents looks human readable, I have checked the rest of the document and everything seems to be fine. Please check this screenshot.

    publii-file-contents-nano

    The only thing I can’t understand, is why the elements declarations shows the path as an absolute path and not a relative one.

    I mean, it reads:

    {
    "path": "/1er-sincronizada-2020/index.html",
    "type": "file",
    "md5": "0f1c47053ce17ef69adf5ee04b63bb76"
    },

    Where the path is /1er-sincronizada-2020/index.html, nevertheless, the absolute path on the server would be something starting with /home/USERNAME/public_html/blog/…

    I am aware that the local paths will be different, based on my preferences and I don’t know if the path is just a part of the full path and the program itself will do something to gather the proper server’s folders information in order to run file checksums.

    If this is not the normal behavior and you think I need to do something to fix this problem with the paths, do you mind telling me where can I set this variable?

    Thanks in advance.

    #3546
    Avatar photo[anonymous]

    Sorry for the late reply – could you check the files.publii.json file with https://jsonlint.com/ to make sure that this file is correct JSON file?

    Also – are you using FTP deployment method? Or other one?

    #3549
    Avatar photo[anonymous]

    Hey Tomasz, good to read you. I hope you are fine and with good health.

    I have validated the files.publii.json file using the link you provided an it resulted in a “Valid JSON” message as you can see in this screenshot.

    files.publii.json_

    For what is about the deployment method: yes. I am using the FTP protocol with SSL/TLS in the Server Settings as shown in the Server Settings page, you can see my configuration on the following screenshots. I have changed some sensible data just for the screenshot to avoid problems.

    serversettings1

    serversettings2

    I am not using the “Use relative URLs” because of the warning shown on the foot of this option, which reads: Please remember: while using relative URLs, some features like Open Graph tags, sharing buttons etc. cannot work properly.

    If you think this option may solve something just let me know. Please.

    Thanks in advance.

    #3554
    Avatar photo[anonymous]

    Now I see that you are using older version of Publii (the relative URLs option is temporary hidden and will be back in v.0.37.0) – could you try to use the newest Publii version (0.36.1)?

    #3559
    Avatar photo[anonymous]

    I have upgraded Publii to Version 0.36.1 as you suggested. The upload process -nevertheless- seems to have only cosmetic changes, that’s all I have noticed so far.

    The upload process started at 19:57 and finished at 20:16 and uploaded every single file on the local folder.

    So I guess there’s something wrong. Maybe something on my end?

    Thank you for your help and patience.

    Screenshots attached.

    Regards

    #3567
    Avatar photo[anonymous]

    @geppettvs – could you share in a private reply file deployment-process.log from Tools -> Log viewer?

    I would like to check if the app recognizes correctly the publii.files.json

    #3568
    Avatar photo[anonymous]
    This reply has been marked as private.
    #3571
    Avatar photo[anonymous]

    @geppettvs – ok, I see the reason of your issue:

    > Malformed files-remote.json file: SyntaxError: Unexpected end of JSON input

    So the app downloads your files.publii.json file, but for some reason it is invalid (it stores it under different name, but it is ok).

    Could you check the content of the input/config/files-remote.json in your site directory? Maybe during download the file is corrupted in some way.

    #3572
    Avatar photo[anonymous]

    There’s no <span style=”color: #ff0000;”>input</span> folder in the server’s blog folder.

    A screenshot is attached.

    How can I fix this?

    I didn’t delete anything, it seems my previous Publii version didn’t create it, and the folders and file does not exist.

    #3574
    Avatar photo[anonymous]

    Sorry – I mean input/config/files-remote.json on your local computer in the site directory 🙂

    #3575
    Avatar photo[anonymous]

    I am attaching a ZIP file with the files-remote.json file inside.

    I can see this file is -somehow- incomplete, truncated at line 1943.

    I don’t know how to fix it.

    Would deleting/renaming this folder make Publii to re-create the file, thus fixing the issue?

    Thank you.

    #3588
    Avatar photo[anonymous]
    [anonymous] wrote:

    Hey Tomasz, good to read you. I hope you are fine and with good health.

    I have validated the files.publii.json file using the link you provided an it resulted in a “Valid JSON” message as you can see in this screenshot.

    files.publii.json_

    For what is about the deployment method: yes. I am using the FTP protocol with SSL/TLS in the Server Settings as shown in the Server Settings page, you can see my configuration on the following screenshots. I have changed some sensible data just for the screenshot to avoid problems.

    serversettings1

    serversettings2

    I am not using the “Use relative URLs” because of the warning shown on the foot of this option, which reads: Please remember: while using relative URLs, some features like Open Graph tags, sharing buttons etc. cannot work properly.

    If you think this option may solve something just let me know. Please.

    Thanks in advance.

    Replace public_html/blog  with  /public_html/blog/ and you should have no more problems. I had the same problem with one of my domains, after adding a slash at the end, it only uploaded the changed files.

    #3596
    Avatar photo[anonymous]

    @kosi77 – it is an interesting tip 🙂 I am not sure if it will help but maybe it is worth to try.

    I have analysed the logs and it seems that on the FTP level the file is corrupted – Publii downloads 57kb instead of 67kb what causes corrupted JSON file.

    And at this moment I have no idea why :/ Because it is a simple download operation using FTP library.

    #3597
    Avatar photo[anonymous]

    The inclined line before and after works very well for all my domains.

    Without a slash at the end, he uploaded all sorts of files every time.

    I don’t know what this is due to, but it works perfectly for me.

    #3604
    Avatar photo[anonymous]

    Hello Tomasz, kosi77

    I have added the slash at the end of the public_html/blog/ and now the process takes less time, yes.

    I initiated the blog sync at 11:44:02 and it finished at 11:45:30, which is much less time than before.

    Now the process shows it will do up to 48 operations which is much less than the 479 it used to do before. And these 48 operations takes less than 2 minutes.

    less-operations

    I have checked the upload process two times and the results and time to upload are almost the same. But the number of operations remains in 48, which I guess may increase based on the new contents or changes I commit to my blog.

    twice

    The files-remote.json now seems not to be broken. It looks fine. So I guess the slash at the end of the physical location is a good solution this time.

    The tip below the field looks fine but it does not warn about leaving the field without the slash. So I guess I didn’t follow the proper instructions when setting my server up. Maybe a warning on the screen may help?

    Thank you very much.

    #3610
    Avatar photo[anonymous]
    This reply has been marked as private.
    #3638
    Avatar photo[anonymous]

    I have created new contents today, in the same conditions I used to have this problem and everything seems to be running fast and smooth this time.

    You can see my blog at https://nuvogue-electronics.com.mx/blog/

    So we can say the solution was to add the slash at the end of the public_html/blog/ as kosi77 suggested.

    Thank you very much. This topic has been solved with your help.

    Appreciation.

    Regards.