#wordpress 2018-11-29

2018-11-29 UTC
[schmarty], [tantek], j12t, jgmac1106, j12t_, [eddie], [jgmac1106], AkyRhO, jeremych_, cjwillcock and jackjamieson joined the channel
#
GWG Hi, jackjamieson
#
jackjamieson GWG: Hi, how's it going? I saw you're on holiday - looks fun!
#
GWG I am suffering for it today
#
GWG But no dev stuff
#
GWG How is your effort going?
#
jackjamieson Pretty well - I've got the improved polling schedule close to implemented. Thinking through how to handle scheduling, given WordPress's cron
#
jackjamieson While I think on that I've been tidying up the code, basically just clearing phpcs issues
[Vincent] joined the channel
#
jackjamieson And parsing is now handled entirely by parse-this
#
jackjamieson For the rest of the week I'm mostly focused on grading papers, so not much dev time until Monday probably
#
[jgmac1106] move to self grading....helps me a ton
#
[jgmac1106] gotta do the same though myself...need to start looking at where folks are
#
GWG I don't know what I will be doing next
#
GWG Probably extra flags in Parse This
#
jackjamieson [jgmac1106]: Self-grading would be nice, but in this case my contrast is just for grading papers, so I've got to do it the old fashioned way! Interesting course though and the papers are good so it's a nice diversion
#
jackjamieson GWG: I'm toying with the idea of using fetching etag headers to tell if a page has been updated since the last poll. Might save a few redundant fetches. If so, I think this would need to be implemented in parse-this. I could try it out and send a PR if you like
#
jackjamieson GWG: But I'm not sure if that would actually help that much. If you have any thoughts let me know, otherwise I'll try it out and see if it makes a difference
#
GWG I have to read up on etags and last modified.
#
GWG And how you think it would work
#
jackjamieson My thought is to store the etag every time I fetch a feed. Then before calling Parse-this->fetch, Yarns will request the page's headers and do a comparison. Then it will only proceed with the fetch if the etag has changed.
#
jackjamieson It's not a high priority, but seems like it could be pretty simple to add so I might just try it out
#
jackjamieson Actually I could do it entirely within Yarns, but if added to Parse-This I imagine it would take the form of a fetch_headers function, or something to that effect
#
sknebel I'd suspect everything that means a request to check the headers might not be worth it in the normal case
#
sknebel but checking the response headers before parsing might, as would conditional requests (include etag or time in the request, so the server can tell you if nothing has changed)
#
GWG I will think on it for a bit
#
jackjamieson sknebel - that's a good point. I think a conditional request would be the best approach. Thanks for the advice.
#
jackjamieson GWG: I'll think about it too - I might just test it out and then let you know if it seems worthwhile
#
GWG I usually do a head request
#
GWG You think a head request is too wasteful?
#
sknebel Depends really. If you always do a HEAD request to check if stuff is outdated, and 95% of cases it is, then it's a waste IMHO
#
sknebel if doing the HEAD allows you to prevent a noticable amount of full GETs from happening it can be good on the other hand
#
sknebel I guess I didn't consider slow feeds properly when first thinking about it
#
sknebel i.e. if your fetch rate matches the change rate of the feed, so you get a changed feed most of the time its probably not helping
#
sknebel but you'd probably fetch slow feeds (as in, posting once every x weeks) still daily or so, there it'd help
#
sknebel so maybe I was wrong overall, and maybe it's worth doing it only for some feeds
[tantek] joined the channel
#
jackjamieson I suspect the head requests are useful in practice, because there are likely to be a decent number of slow feeds. I'll do some more reading about conditional requests too, and maybe try to implement a test
#
jackjamieson sknebel++ Thanks!
#
Loqi sknebel has 6 karma in this channel over the last year (91 in all channels)
#
GWG I do head to determine content type
[jgmac1106], [schmarty], [dave], jackjamieson, [tantek], [eddie] and jgmac1106 joined the channel