Fluke 8840A Faulty CPU

[essele]

And bus writing now also works :-) ... I can initialise and write to the display, and once I stopped testing with an 8840A that seems to have a broken display everything sprang nicely into life -- so it looks like this is really going to be possible. The timing all seems ok, at least with the keyboard and display controller and the EPROM, it may be a little more challenging with the ADC, I'm not sure if the reads and writes are clock dependent.

I love these PSoC devices .. that extra capability to do some clever stuff at the hardware level really makes a big difference. With my current implementation I'm at 63% UDB usage, but that's with zero optimisation, and there are quite a few things I could do if needed. It's looking like everything else can be done with software, so this shouldn't be required.

Now to focus on the translation of the firmware, there do seem to be a few register indirect accesses to firmware addresses presumably for lookup tables or preset data, so these will take a bit of manual effort ... the rest I'm hoping can be fully automated, and then I can worry about tidying later.

[essele]

A quick update...

I've spent quite a lot of time on automated code-path discovery so I can track RP, tests-followed-by-conditional-jumps etc, all with the aim of being able to automate reasonably efficient code generation, however I'm going to have to put that on hold for a bit ... there are quite a few nasties in the code that mean I need to rethink it slightly ... things like popping an extra return address off of the stack so you don't actually return from the CALL you think you should, and pushing FLAGS so you can use an IRQ handler (and IRET) as a normal CALL.

I still think this is the best approach, but I need some time to work out how best to code it ... my current solution doesn't revisit already travelled paths, which then makes it virtually impossible to properly track stack usage.

So, instead I've built a basic CPU emulator for the Z8, which should hopefully be good enough to prove the whole principle. The basic emulator is done, I'm now working on translating the register reads/writes into PSoC related code to simulate the same behaviour, then I'll need to figure out the interrupt side of things. It wasn't until I was well into this that I discovered there is also an emulator in MAME ... that was very useful in checking my logic. The MAME one isn't really focused on speed, so I've had to approach things a little differently.

I've discovered a few things that need some work...

1. The power line frequency is determined by counting how long P20 stays low, this is a tight loop and therefore dependent on execution frequency, it would actually work for me as is (since 50Hz is the slowest), however I'll need to think about how to address this for 60Hz and 400Hz line frequencies. The simplest way would be to adjust the compares in the firmware, but it will be interesting to see how far off the performance is by default (I can do an isolated test for this) and I'm not really sure how I can test 60Hz/400Hz ... maybe I can test outside of the unit and inject a signal.

2. Timer0 is used as the baud rate generator for the UART for the GPIB board. This is only ever used at a fixed 31250 baud so that can simply be hardcoded into the UART for now. It's relatively easy to translate the prescaler values into PSoC baud rates, so I could get a bit more flexible (not sure if the 8842 uses a different rate?)

3. Timer1 is used to set the ADC sampling rate, these vary depending on line frequency and slow/med/fast speed setting. So I've added a PSoC PWM module to output the timer and can translate the prescaler/initial-value settings into PSoC friendly calls.

4. Most annoyingly it seems that the ADC reads are all done using 'extended bus timing', this is a slower version of the bus interface that adds cycles to allow for slower devices. Interestingly the datasheet says it adds two cycles to the DS phase, however the timing diagram only shows one. (Did nobody ever proof read these things? There are loads of errors!) I am going to have to revisit my Verilog component to cater for this, at the moment I don't have a free control bit to switch modes, but I have a way to work around that.

5. On the interrupts ... we have keyboard, ADC, serial-in, ADC-timer ... two aren't used - the other timer (serial baud) and serial output. Serial output appears to be polled. I don't think there are any real issues here, I just need to figure out how to translate the whole priority side of things and make sure the behaviour of the various registers is the same.

Lee.

[TheDefpom]

I am genuinely impressed by your project, and I thought I was doing well decoding the ADC a few years ago in one of my first YouTube videos whilst fault checking a 8842A

[essele]

Thanks DefPom ... I'm actually really enjoying this, it's a throwback to some assembly stuff I did when I was 16 (and that was a very long time ago!)

I've just done the first performance tests and things are actually looking reasonable. I've recreated the code that does the power line frequency checks, so it's pretty simple and doesn't really involve any complex instructions, so optimising further may well be a bit of a challenge. However...

As I mentioned above the code goes into a tight loop incrementing a register pair until the power line signal changes, so I'm assuming this is timing the 'off-time' of the  'freq ref' line from the power supply, so the number increases as the frequency decreases.

The firmware looks for the following (if I'm reading it correctly):

1. If it's less than 0x200, then we assume 400Hz supply.
2. If it's more than 0x470, then we assume 50Hz supply ... I assume this would actually be the 55Hz point, well that's what I would use.
3. Otherwise it's 60Hz.

Bearing in mind I'm on a 50Hz supply at the moment, I can only currently read the 50Hz value, but I've extrapolated based on that, and I get the following:

For 50Hz I get a value of 0x578, with optimise for speed, I get 0x5c8.
For 60Hz that extrapolates to 0x48e, and 0x4d1 with optimisation ... so this would miss and be assumed at 50Hz.
For 400Hz it would be 0xAF, or 0xB9 with optimisation.

So it's actually pretty close ... obviously you would want to be well inside the values to have some margin to play with, but whilst it means that this bit may not function it shows me that the emulator is actually running at a reasonable speed. I may well have to resort to cycle counting to have a better estimate of what the actual Z8 values would work out to be ... I can then look to see if I can further optimise .. but that's a job for later.

The other thing is that it proves that my external port access to port20 is working ... and that was just two lines of code!

EDIT: I'm a muppet ... I've just been trying to work out the actual cycle count and realised that actually the emulator is running faster than the original Z8 ... it's bloody obvious because my numbers are bigger than the expected ones, therefore I manage to go around the loop a few more times! That's really good news, it's much easier to slow it down ;-).

So the Z8 manages to go round this loop in 128 clock cycles (8Mhz), therefore it can do it 62500 times a second. I'm managing 70000 and 74000 depending of whether I'm optimised for speed! So it looks like I'm between 12% and 18% faster ... these are simple instructions though, I'm sure once we're in to some of the maths ones with loads of flags this won't hold.

[essele]

I've now implemented the extended bus timing (but only for reads, since it's only needed for the ADC reads), the datasheet was actually correct, I was mixing clock cycles, machine cycles and the various other cycles they seems to have made up for the Z8.

I can read the 5 values from the ADC and they all come back and seem to be 6 bit values which is a good sign ... the actual values are a bit questionable, but I'm currently not reading them straight after the interrupt, so they could be a mix of samples or values mid-sample. I've got a plan for the interrupt mechanism and I'll use this as the first test.

I was looking again at the performance numbers ... actually I think I had misunderstood the performance of the Z8 and it's four times slower than I had thought...

The external clock (8MHz) is divided by 2 to give you the internal clock (obviously now 4Mhz), then the instruction 'cycle counts' seem to be "timing states" which pretty much seem to take 2 clock cycles each anyway (although some of the timing diagrams show it as 3.) So that actually means that an instruction taking 10 cycles on the Z8 is really 40 cycles at 8Mhz, which then gives us 240 cycles on the 48Mhz ARM to emulate the same behaviour.

This is born out by the tight loop for power line frequency detecting ... that was a 32 'cycle' loop (INCW, TM, JR), but actually working it through assuming I was right about my 55Hz cut-off we had 62500 iterations per second, which equates to 128 cycles at 8Mhz per iteration, so for 32 'cycles' on the Z8 that means it's effectively divide-by-4. So it's really a 2MHz CPU.

I've figured out a nice way of handling interrupts that doesn't require me to check something each cycle, and I think I've worked out how to map the various interrupt request, mask, and status lines.

I still need to work out how to deal with the Z8 reset line ... it's a normal GPIO for the PSoC, but since there's the potential for getting stuck with interrupts disabled I don't really want to handle it as a normal IRQ ... I think the watchdog reset might have potential here. I will also experiment with using another pin wired directly to the PSoC reset line, that way taking that pin low should reset the device, I can then optionally internally connect that pin to the Z8 reset line to give a configurable reset (experiments to follow -- not sure how often this is actually used during normal operation, presumably it's just an error condition.)

So next steps...

1. Interrupt mechanism + mapping interrupt registers.
2. Finish off the port configuration registers (inc. serial and timer configs.)
3. A debug exception mechanism for handling illegal instructions and register configs I don't deal with (external stack for example.)
4. Mechanism for timing instructions to see how close to the Z8 we can get.

Then I should be on to actual firmware testing -- exciting.

If it works then I think it will be worth building a board to try to extract the internal firmware from the Z8 so that we can get at the different versions, I only have v2.3 for the 8840, and I've got a working Z8 and EPROM that seems to be v2.5 that I'd love to extract.

[jaromir]

Great work. I'm really impressed by your progress so far.
Is there a chance you could release your work to public? I'm pretty sure that apart from being useful to folks with broken meters, it could also serve as source of knowledge to those ones being interested in how things work.

[essele]

Hi Jaromir,

Absolutely -- I fully intend to release this all once I get it working. I'm starting to put things into github, and I'll switch it to public once it's in a usable state.

Lee.

[essele]

Huge Progress...

- Implemented an interrupt mechanism (although more on this later)
- Finished off the port control stuff, and all the register read/write translation
- Added a debug mechanism to trap some stuff that shouldn't happen
- Fixed decrement-and-jump-if-not-zero instruction so it wan't increment-and-jump-if-zero (I must have been having a bad day)
- Fixed a few other instruction issues
- Banged my head against a brick wall for quite a while

... and ...

It works ...  mostly. It boots up, will accurately read DCV, DCA and Ohms, autorange works, front/rear selection works, it detects the lack of AC board (haven't tried that yet), the self tests will run (and pass) ... the calibration looks ok, so I suspect it's properly reading the existing cal ram.

How pleased am I!!!

There's still some work to do ... to get this far I've had to do some stuff in the interrupt handling that I shouldn't really have done, so I need to understand that a bit more, also some of the small buttons on the right cause an exception where it's trying to execute code at an address that doesn't exist. Then there's the serial stuff to check for the GPIB card.

I don't think the interrupts are working properly ... I don't see any timer interrupts at all, although that doesn't stop the basic functionality, the rate button doesn't do anything at the moment, so I suspect it might be related, and the execution of dodgy code does highlight I've got something else wrong somewhere, so once I find that other things may start working ok.

QUICK UPDATE:  I've fixed the bad code issue (was a mistake in the register-pair call code) so now all of the small buttons seem to work, I can change the rate, selected external trigger etc.

[TheDefpom]

nice work, well done.

[Johnny10]

Essele your talents are showing!

[coromonadalix]

Wow  im totally impressed by your work 

[essele]

Thanks for the compliments! I am feeling quite pleased with myself.

Anyway ... I've reworked the interrupt code and it seems a lot better, all of the buttons are more responsive than they were. I've tested the AC board and that all works perfectly ... which is good, because that's partly where this all started ;-)

I've done some side-by-side testing with a standard Z8 based unit and I can't see any difference.

Just the GPIB board (and hence the serial connectivity) to test now ... I need to get my USB GPIB adapter working on a bit of normal kit first so I know what good looks like. Oh, and I still should probably do something with the reset input.

If anyone feels like donating an 8842 with a faulty CPU , I'd love to get that working as well ... it may just work (with appropriate firmware) but there may be features it uses that I haven't implemented. (Just make sure the screen is ok, I wasted quite a few hours on an 8840 that turned out to have a broken VFD.)

[coromonadalix]

If your substitue cpu work,  can you add an serial port instead of the gpib ?? or an usb "rs232" port too ??

Maybe a mix of this project ??
https://www.eevblog.com/forum/projects/ar488-arduino-based-gpib-adapter/

thks


Would be nice to have a 8842 donnor ...

[essele]

Hi Coromonadalix,

The CPU communicates with the GPIB card using some kind of serial protocol, so there's no reason I can see why you couldn't develop a different card that provides an external serial or USB port if you wanted to. You would just need to reverse engineer the protocol which I imagine could be done fairly easily by sending individual GPIB commands and then tracking what internal serial comms happens.

You could also hack the firmware to send/receive serial data straight out of the unit, or you could add extra routines in the PSoC to so something similar, but either way that would require a lot more firmware understanding to know where to pull the data from and how to trigger state changes. It's definitely doable, but not something I'm planning on looking at.

The other interesting thing that could be done is to build a completely different interface to the display, this display output code is really easily found in the firmware and could actually be trapped quite simply in the PSoC (it's just writes to the keyboard/display controller) ... so creating something that would drive an LCD for example would be reasonably straightforward. Since I have a machine with a broken VFD I might consider this once I've got everything else working.

[coromonadalix]

Hi  essele

You have some 3.2" oled  who where used to convert an hp34401a vfd ...   made by qu1ck

https://www.eevblog.com/forum/repair/hp-34401a-dmm-with-leaking-segments/msg1572961/#msg1572961

[essele]

Further quick update...

I've been working on the GPIB side of things ... spent quite a bit of time trying to understand why I wasn't seeing any serial response from the card at the PSoC, despite being able to see the data at the card ... turned out to be an almost invisible solder bridge between the serial-in pin and GND.  It's amazing the things that burn hours!

I've done some further work on the serial emulation and there is some progress, but it's not quite right yet.

The card is detected, and I can press the local button and I get presented with the GPIB address, so there is a level of working comms. However I see some spurious behaviour which looks like the unit is responding to random/corrupted commands (although it appears repeatable on each power cycle) ... my guess at this point is either reception problems due to framing/baud mismatches, or problems with IRQ handling.

Timing looks ok on the scope, and matches the timing diagrams in the manual.

The job for the weekend will be to add some framing/overflow/underrun detection to rule that in or out, and then to re-think the way I'm doing serial interrupt handling. Fingers crossed this will all be done in the next few days.

[jaromir]

Great job, great post! Keep them coming, please.

[essele]

More battling with the serial side ... I've checked the GPIB board in a standard 8840 and it works fine, I've check the PSoC CPU in the working 8840 and that behaves exactly the same way, so I've eliminated any issues with the board or other parts of the 8840.

I was slightly worried about the timing accuracy (as per comments way back), so I tried a few different ways of generating the clock, but with no difference to behaviour, even with quite a major variation in frequency, so it doesn't seem to be that.

I've now got a protocol analyser (Analog Discovery) hooked up and I've looked at the serial rx/tx lines for both the working unit and the PSoC based one from power up ... an example of a small section (where there is a difference) is shown in the photo, the top sequence is the working one.



The data is pretty much identical, but what changes quite dramatically is the timing. I did implement one fix which actually slows down multiple byte transmissions, because the PSoC was much quicker, but that didn't change anything.

So I think there are a couple of avenues to still explore:

1. The strange lack of timer IRQ's ... there is a full ISR in the firmware but I never get a valid IRQ trigger for it. I need to recheck this, since it's doesn't feel right to me. I'm not convinced it's this problem though since that would more likely result in a lack of something happening, rather than something happening too quickly.
2. IRQ handling again (although I have redone this a few times now), but it could be a prioritisation thing where I'm allowing the receive IRQ when I shouldn't be, or a window with enabled interrupts that shouldn't be there. Again I don't think that's the case, but I will go through it again.

This has been a really interesting journey into lots of different things ... but I do want it to end soon ;-)

[essele]

I'm not sure I'm making much progress on this.

I've reworked the IRQ system yet again ... and I see exactly the same behaviour whichever way I implement them, so I'm fairly sure that the overall mechanism isn't contributing to the problem.

The timer ISR looks to be fine ... my interpretation of the code is that the ADC irq drops through into the timer ISR when needed (which makes sense given it's the timer that triggers the ADC), so the timer ISR is only likely to be called as an interrupt in specific circumstances, perhaps when external triggers are used?

I have three things to follow-up on now:

1. The two pin based IRQ's (ADC and Keyboard) have slightly different behaviour to the Z8 version because they share an interrupt on the PSoC, so I'll rework them to use DSI routing and individual interrupts. I really don't think this is the issue.

2. The serial system is slightly different because of the FIFO and the way you clear the IRQ, so I'm actually considering building a UART (at least the RX side) in the digital logic to more closely resemble the Z8 version. Again I don't think this is the issue as I think I understand the way the code interacts and I can't see how it would cause a problem.

3. The other option is that I still have a misbehaving instruction which is only used (or only a problem) in the serial code. So I'll go through the implementation again checking everything, but it does work for short period of time, which makes it feel more like a timing/IRQ issue than an instruction problem.

EDIT:  And potentially there is: 4. From re-reading the Z8 manual it seems that the interrupt mask register can't be updated unless the DI instruction has been used to disable interrupts, you can't simply write a 0 to bit 7, my implementation doesn't enforce that. Then the manual isn't clear if writing a 1 to bit 7 is ok to enable, or you have to use EI. There is a chance the firmware relies on these controls, so that's a fairly easy combination of things to test as well.

Lee.

[essele]

So I’ve implemented independent IRQs for the keyboard and ADC, no major change in behaviour, but it feels better as it’s more closely aligned to the Z8 approach.

I’ve built a UART, well actually a UAR, in Verilog ... surprisingly easy if you skip most of the error conditions ... and it seems to work ok and more closely mirrors the Z8 approach. Again no significant change. (As an aside, I never thought i’d be building UARTs in Verilog ... how cool is that???)

I’ve also rechecked all the instructions, put a load more debug in to check for strange things, and tried all the variations of EI and DI control, and nothing seems to impact the behaviour.

However ... it’s now very consistent and repeatable, so I think some of these changes have helped resolve issues.

I have then also started looking at the timer implementation since I was a bit too quick to dismiss some of the intricacies ... I’ve redone this with a proper down-counter with correct behaviour at re-load (which took some more Verilog and a reversion to the old UART for space reasons for now.)

The upshot is that I now have a rock solid external trigger (with GPIB board) setup, but there are still challenges with local trigger if the GPIB board is connected ... I’m fairly sure it’s timer related, so some more work is needed - but it’s time for a weekend of well earned gin.

Lee.

BTW - I’m intending to publish the git repository on Tuesday, maybe others can help get this over the line.

[essele]

Ok .. I've had enough for a while on this, I just can't get over this last issue.

I've even implemented a more efficient serial receiver using a datapath component so that I can fit it all in to the programmable logic, but no change in behaviour, so I think the serial side is actually a red herring (especially given the external trigger, which is over serial, seems to work perfectly.).

For the time being I've just dumped all of the code, schematic, and pcb stuff into GitHub ... I haven't tidied it much, so there are lots of remnants of debugging in there and still quite a few things to tidy.

https://github.com/essele/Fluke_Z8_Emulator

If anyone feels like getting stuck in then please let me know.

[SilverSolder]

I noticed this sticker on your meter:




 

[coromonadalix]

yep  broken sticker  loll

Keep up the good work,  hope you find the little buggar ..

[essele]

Ok ... I’ve just come back to this as I need to sort out the three broken 8840A’s that I now have, all with CPU issues.

So now I’ve implemented a firmware ‘patching’ mechanism by using an illegal instruction as a special emulator call and then patching this into the firmware in some key places, this makes it easier to do the following:

1. Replace the line freq code with native code to ensure accuracy.
2. Replace the main delay loop subroutine with a more accurate one.
3. Put test code in at specific points to better understand what is going on.

I also redid the interrupt code yet again, and spent ages decoding serial again ... both of these to no avail.

However, using (3) above, I was looking more closely at the timer, and for some reason I completely missed the fact that it sometimes configures itself to use an external clock input (the ADC signal) rather than the internal clock ... this seems to be when using external triggers. My emulator doesn’t currently support that and it just continues to run with the internal clock.

So my theory now is that, whenever you have a GPIB card, trigger signals are generated by that instead of the normal clock, and hence this then uses the timer differently. So I think with my current implementation the GPIB triggers and the timer triggers are interfering with each other ... which fits quite well with what I’m seeing.

So, I’ve reworked the timer using two cascaded TCPWM modules and I should be able to remove my custom verilog hack as a result. I just need to add support for the external clock input (which looks straightforward) and then I have a good level of confidence I might have cracked it.

Hopefully will update at the weekend...

[essele]

I've finally cracked it!

After much probing and reworking, I've managed to get it working...



I reworked the timer code, I put in "patches" to accelerate the ADC reads and the display updates, so they are handled by the PSoC and not emulated, I put in a patch to more accurately handle delays, I even put in a patch to cache the cal EEPROM to speed up that access. I also used another timer to delay the serial send and receive. None of that worked.

So, I redid the interrupt code again (probably for the 10th time now!) and now have it so that it runs with PSoC IRQ's enabled, the handlers just set an appropriate flag and then the main execution loop checks to see if an IRQ has occurred and causes the Z8 ISR to run.

That didn't work either ... at first!

So when I moved the IRQ check and ISR call so that it was after the "next instruction execution" everything started working, so I think the Z8 will actually check the interrupt status during a given instruction execution (for example the one after the IRET) and I was checking before, and therefore potentially jumping into another ISR without executing that instruction.

So everything is working ... serial comms to the GPIB card (although I still need to check GPIB actually works), remote triggering, all the self tests, running on slow, medium, and fast speeds. All looks good.

It does break again when you compile with any optimisation enabled, so there is likely some very tight timing needs, but "None" seems to do the job nicely and I'm not inclined to spend any more time on this. I'm going to get my three 8840A's up and running and then I never want to see another one ;-)

I'll update the GitHub once all is complete.