Test Equipment Anonymous (TEA) group therapy thread

[bd139]

I guess I missed this morning's excitement with broken and duplicate threads and messed up post counts. Did we lose anything or was a restore a success? 

Yeah, I'm late to the party as well. Apparently, the forum was restored from last night's backup. Dave created a thread at https://www.eevblog.com/forum/news/news-forum-reverted-to-backup/

@bd139, is your TEA thread backup script still running? Coincidentally, I was thinking about it the other day.

Ah, just saw your post from yesterday:

Nice one 

I really need to finish my archive scripts off. It's clear there is a risk that we lose everything.

So, bd, in what format are you storing the thread contents? Since I have domains for TEA, now, I can put up a read-only archive/mirror.

It's being scraped with selenium/python into a SQLite database. My Selenium-foo is shite so it's taking a lot longer than anticipated. This will pull ALL external images which are linked and archive them as well.

SMF's absolute shit garbage HTML is causing me some grief as I need to cope with about 2 million horrible things it does.

Intent is to render it out of the SQLite database again using Go/template as static html. It can be extended and regenerated then to do things like text search / tagging etc later rather than be a flat HTML archive.

Recent issues suggest we should probably just wget the whole thread though as that's got a shorter turnaround on it.

Edit: also I have to work around things like SMF throwing DB connection errors every two minutes 

It'd be nice to get a dump of the relevant tables but I get the feeling that wouldn't go down well if we asked 

[bitseeker]

Yeah, I wouldn't want to make that request and have something go wrong with the forum as a result. Just wget-ting the thread at a nice leisurely pace should be OK for a basic backup. I'll give it a shot.

[bd139]

Yeah that’s possible based on today 

I’m going to bed now but let us know how you get on with wget 

Edit: looks like there’s a couple of scrapers out there already as well. As always I didn’t go and look first:

https://github.com/aayushagra/smf-scraper

https://github.com/PoLaKoSz/SMF-Scraper

[bitseeker]

wget works OK, but isn't ideal for incremental use because SMF appends the current PHP session ID to some URLs, which then makes some common files look unique every time wget is executed.

But, for now, I grabbed the ten most recent pages, then started it on the full thread. In order not to hit the server too much, I have it wait a couple seconds between fetches. Unfortunately, wget doesn't have an option to wait only between fetches to the same domain, which would be faster since not all elements on a page are from eevblog. At least it supports modification timestamps to avoid re-downloading files that haven't changed.

Anyway, it's been chugging along for a couple of hours and will be done in another 16 or so hours (all pages of the thread with all their requisite elements).

[Brumby]

Ah... I can post again.

None of the actions which would modify the database were available for me this morning... until now - when I have to leave for work.

*sigh*

[beanflying]

Ah... I can post again.

None of the actions which would modify the database were available for me this morning... until now - when I have to leave for work.

*sigh*

I seem to have had a productivity gain today - until now 

[bitseeker]

 

I was trying to be productive, but wget needs babysitting. At one point it got stuck waiting to connect to a site. So, I had to terminate it and set a 10-second connection timeout and only two retries (default is unlimited ). Later, it got to a site that successfully connected, but nothing after that (like a tarpit). It sat for 15 minutes before timing out. So, I killed it again (30 minutes is too long!) and set the read timeout to 10 seconds, too, and added the offending domains to the exclude list.

Each time, I have to restart from the beginning in order for the PHP session ID to match up and to enable wget to rewrite all the links properly when it's done. Fortunately, it doesn't take long for it to catch up where it left off since it skips unchanged files (i.e., all the attachments).

[mnementh]

Yay, we're back; I'm going to bed.

mnem
 

[bitseeker]

Night! I should hit the hay, too. Hopefully, this'll finish archiving in the meantime. Keep your fingers and claws crossed. 

[mnementh]

Claws, wings, toes and tail.

mnem

[mansaxel]

The small Bulgin mains connectors.

My Solartron DVM has one of those. Horrible connector, truly bad. Unshielded quick-connects would be worse, by a little bit.

[PA0PBZ]

For those who missed all the fun: When this showed up in the 'Show new replies to your posts' my first thought was that the forum finally choked on this thread.

[med6753]

For those who missed all the fun: When this showed up in the 'Show new replies to your posts' my first thought was that the forum finally choked on this thread.

I was on very briefly Sunday morning and I saw that plus Dave's admission that he screwed up.  Decided at that point to log out and didn't get a chance to get back on until late Sunday and saw that things were at least somewhat back to normal. 

[tautech]

For those who missed all the fun: When this showed up in the 'Show new replies to your posts' my first thought was that the forum finally choked on this thread.

I was on very briefly Sunday morning and I saw that plus Dave's admission that he screwed up.  Decided at that point to log out and didn't get a chance to get back on until late Sunday and saw that things were at least somewhat back to normal. 

Except for a ship load of missing posts ! 

[mansaxel]

I've tired to use several of the other offerings, Plex etc. and found them wanting like you have. I settled on Kodi precisely because it isn't a pain in the backside and would deal with media I already had on servers and was trivial to hook up to my chosen off-air source.

Kodi did not attract me at first, because I was more looking for a server solution. I think I need to reevaluate that.

[bd139]

I ran Kodi for a bit on those Lenovo M600 boxes. Alas it's too complicated and buggy so I rather prefer to just rip shit off and use a USB stick and the TV. I don't spend enough time in front of the rot box to justify it really

[mansaxel]

I just realised I've got a slightly older IdeaCentre left over. Might do something about that. I really like the tuner Cerebus posted about.

Edit: This is all about being able to ship off the old TV, and not have a rootkit replace it. No modern TV with a DVB-T2 tuner in is free from the spyware that turns into abandonware and more or less bricks itself on a 4-year cycle. Currently looking into digital signage displays to get something unencumbered.

[Specmaster]

I did like Kodi at the start, but once I found out how often I'd have to search for replacement channels, I gave it the old heave ho.

I have just ordered a refurbished Humax FVP 5000T from Humax Direct, comes with a full 12 month warranty and according to customer reviews, is indistinguishable from a new one and if true will be like my Nextbase 512GW dashcam, also refurbished but as new including the same warranty, "winner winner chicken dinner"

The Humax cost me £109, free delivery and was only a few more £ than they are selling for 2nd hand on eBay with no warranty, so I thought its a "no-brainer" and should keep SWMBO happy once the VM boxes get returned.

Thanks everyone for your ideas and recommendations, much appreciated   

[FransW]

Kodi comes with joy and leaves with a headache.

[bd139]

Sounds like my ex wife.

[BU508A]

Sounds like my ex wife.

Have you checked her second name? 

[med6753]

Sounds like my ex wife.

The headache and memories there of do eventually go away or perhaps I'm going feeble minded. Whatever it takes. 

[beanflying]

Get a Puppy/Cat or buy excessive old Test Gear - working for me and I don't currently own a Dog or a Cat

[McBryce]

Sounds like my ex wife.

Sounds like every ex-wife.

McBryce.

[Cerebus]

I really like the tuner Cerebus posted about.

It attracted me because for years I've used a couple of EyeTV tuners (USB connected). They didn't handle DVB-T2 modulation and weren't the most reliable thing in the world and so it was time to move on from them. I looked at PCIe capture cards and looked at the stack of now useless  old ISA and PCI (non-PCIe) cards for various things that I've got sitting around and then a friend pointed me to the HDHomeRun and I didn't look back. Given that twisted pair ethernet has outlasted about 4 PC bus standards it looked like a no brainer in obsolescence terms. Moreover, disparate and separate things can talk to it from different machines to get access to the airwaves which leaves flexibility on what software I use with at, and the protocol it uses to select and stream channels and multiplexes is documented well enough that I could write my own access layer with relatively little pain should I need to. I was sold. 

As long as you only need to access 4 different multiplexes simultaneously you can pull as many program streams off it as you like in theory. In practice you're limited by is the 100Mb ethernet on it to either around 10 simultaneous HD programme streams (~8Mb/s each) or around 20 SD programme streams (~4Mb/s each), or some intermediate mixture. There are apps for all the usual tablet and phone platforms that let you use it to simultaneously stream live TV as well as using it with some sort of  more technical central PVR solution like I do.

The HDHomeRun has been on the network now for a couple of years and has proved completely trouble free. It's had ongoing firmware updates on a comfortingly regular basis. Its coped with my relatively crappy antenna with no issues.