Project isolation, social islanding?

[watchmaker]

I have a starter kit sitting on my shelf.  My EE/IP attorney son in law told me they were crap for learning and product development.  So there it sat for over a year.  I may find use for the modules as shortcuts at some point.

The issue is they are marketed for learning and users think they understand because they have success.  Kind of self reinforcing.

But what have they understood other than programming?  Thevenin?

Arduino and such reveal a pent up demand for true educational kits.  When I was a kid, there was a subscription based program (look in 1960s Pop Elect) that provided tube based projects with 120vAC on cardboard bases.  It is no surprise these disappeared, (more so that I survived).  But they were popular and fed demand for “at home” learning.  I think that Arduino takes advantage of that demand.

I just started the Elenco AM/FM radio kit.  Unlike the Heatkit/Eico kits of the past, this kit does more than provide solder practice.  It has sections devoted to theroy of operation and to using a DMM/VOM, sig gen and scope to analyze the various circuits.  For example, max power, BW, distortion, etc are analyzed for the audio amp; there are a  LM 385 ckt and a discrete AB amp module to compare.  It is not complete but it provides a platform for further experimentation and learning.

I will write a review of this kit in the sandbox (Beginners) when I am done.  (FWIW, I am just finishing capacitors and opamps in my program based on Boylestad and Real Analog).  Doing the kit at this stage helps me focus on why the very basics are important to understand.

Kits like this motivate the learner to learn about the circuits and device datasheets in more detail using the internet.  The kit has given relevance to the subject.  It provides a structure for combining theory with analysis.

I think there is an opportunity here for one of the more entrepreneurial members of this board.

The problem with people thinking they “know” is not unique.  It is very common in watchmaking.  Electronics seems to be another example of where you cannot know until you actually learn it in your hands.

[Siwastaja]

Yes. Trying to make someone understand why arduino or platformio is crap, or why they have failed as educational tools because they have never tried to create a product with them is pointless... because they are not making products, but contraptions.

There was a time, maybe around decade ago, when some people really implied or point-blank claimed that Arduino is now the pinnacle of professional electronic product engineering. I was actually questioned why I don't want to use Arduino in my designs (many hobby back then, but many also serious) like I owe an explanation. That was weird as fuck.

I might have explained it by showing how slow and bloated digitalWrite() is, or how crappy the pin header layout is (impossible to use either soldered or solderless breadboards, quite crucial for actually building hobby stuff or prototypes), or simply that because I had been using very same AVR chips for well over a decade already with no problem why should I downgrade?

These arguments had effect of me becoming some kind of "Arduino hater" when in fact I was just responding to so called stupid questions. I gave up. Since then I simply referenced a marketing claim from Arduino's own website: Arduino is meant for artists. Probably that one is gone now, but I think it's a good point. Listen to how they themselves describe their product. Use if you find it useful, and where it suits. Arduino ecosystem is especially suitable when you find the combination of modules or "shields" that is suitable to you. If you are developing something for which no "shields" exist, then the value is pretty limited and the whole Arduino thing only becomes a nuisance. The whole point of it is that it's easy for someone with very little experience in either electronics or programming to use these building blocks and software libraries. But if you don't need it, then you don't.

[paulca]

Apparently it's not just "hobby projects".

Apparently I don't communicate well in my professional domain either.
https://www.reddit.com/r/SoftwareEngineering/comments/1f9iuu0/modern_architecture_and_management/

It must be me.

In work my manager told me that the functional design specification document was not intended for development to use as a blueprint.  When I protested that, no, that is exactly what a functional design specification document is for.  He told me in paraphrased words.... "Stop bringing your prior experience and knowledge to the team.  Accept what the team and doing, embrace it and get in line."

Politely, as a qualified software engineer, you provide service to me. KNEEL mortal!  and "No".  My profession is worth more than my job.  This is a government project.  I can be subject to negligence charges.  The company insurance policy does not cover that, nor does it stop my going to prison.

What I did do, which thankfully restored my faith somewhat in humanity is run things by the department head.  He confirmed that I am not insane, there are serious issues.  So... my balls are well and truely out on the vice.

The way I look at is this.

Working software and money.  It is only when both of those are satisfied and delivered can be talk about bonuses like making friends.
If I die on this hill.  If I die defending the engineering position, it will be service to my fellow engineers, junior and senior.  It will be an honourable death.
I'll probably get a pay bump moving to a new company anyway.

On a more personal note.  I won't lie guys.  I am struggling with this.  I think deleting my reddit account would be a start.  On the career and profression I need to move jobs to somewhere that values engineering values and doesn't suppress them with business objectives.  I know they exist.  The trouble is... market is dead right now.  Nobody is investing, people are being laid off.

[paulca]

Is it the tendency towards "straw-manning" which is catching me out on communications?  Am I exposing too much "tinder" for haters by doing so?  Too much analogy leaving open the interpretation such that those who don't want to help, but want to further their own ego can troll you on details and pedantry?

My intent is to use analogy to simplify.  To create a model that anyone can construct in their head.  Such that discussions can proceed on the topic.  Actually... this level of thought abstraction challenges many it seems.

If I am asking questions, should I refrain from presenting straw-men in fear they be burnt for unexpected and usually irrelevant reasons?  In the previous reddit example, I got berated because of something which came down to "granularity", not dividing workstreams outs.  I also have no idea what they are talking about 1 service = 1 team.  Those are either very large services, very small, non-productive teams or I am completely missing something.

In my defence?  A straw man, in my opinion is meant to be burnt.  People who send business emails with pink, or orange email signatures might disagree.  It's a minimal effort attempt at defining or documenting a much more complicated issue.

On reddit, if you raise a straw man about a tool or technique, it seems as thou the majority are more interested in burning it and giving comments like, "It was flamable, made of straw, I set fire to it HAHAHAHALOL." or "You shouldn't dress your straw men in plastic shell suits, a single match can burn it in seconds."

Maybe I am just being completely and utterly stupid in expecting any kind of sensible answer, discourse and response on places like reddit.

I HIGHLY expect that almost ALL successful software engineers avoid the place because they are too busy in the real world and realise that place like Reddit are not the right forum.

With virtually ever software engineering project (electronics projects?) being strictly under NDA.  Is it any wonder the actual professionals are quiet and the responses I get are not measured or professional?

Just like my day job, everyone seems to busy accepting the crap to have time to discuss making it better.

It seems to fit the mould I should just join them.  Isolate myself and talk only with my project peers.  My trouble there is, my project peers are also full of it.

[ebastler]

Maybe I am just being completely and utterly stupid in expecting any kind of sensible answer, discourse and response on places like reddit.

I HIGHLY expect that almost ALL successful software engineers avoid the place because they are too busy in the real world and realise that place like Reddit are not the right forum.

I have never been active on Reddit or followed any channel there. But whenever I end up on some Reddit page as the result of a Google search, I get exactly the impression you describe. Most users there seem clueless, careless, just there for the trolling, or any combination of the aforementioned.  There may be parts of Reddit which are different, but if they exist, I have never found them.

StackExchange seems like a much better place for software-related technical questions; maybe others here have further recommendations?

[nctnico]

In work my manager told me that the functional design specification document was not intended for development to use as a blueprint.  When I protested that, no, that is exactly what a functional design specification document is for.  He told me in paraphrased words.... "Stop bringing your prior experience and knowledge to the team.  Accept what the team and doing, embrace it and get in line."

To me that would read as "go find another job after dinner". This must be some kind of UK specific thing though as posts with this kind of weird workplace shit have a common demoninator that they are (mostly) posted by people from the UK. Maybe look abroad for a job and move there. No, I'm not trying to be pedantic here; just friendly advice.

[John B]

No specifics on the original post in terms of the project? I would interested to hear about it.

I've been dabbling a bit in Home Assistant, also getting a feel for the overall project direction, and the community.

I've put a lot of effort into OpenHAB, I think I'm going to continue with that for the time being. But the time investment is not the only reason choosing it over HA.

I have some impressions/observations about the HA project and community, but I am curious to hear why/why not HA is/is not suitable for your needs.

[paulca]

No specifics on the original post in terms of the project? I would interested to hear about it.

The irony it's rather "simple".  It's just a "layer down" from HA et. al.  It's all code based. 

To be rude to it, it's a bunch of python scripts hanging off an MQTT bus.  More of the work behind it has been in avoiding code by using clearly defined processes, message structure, topic naming.  99% of boiler code has been rolled up in to the library and a "stub" include which even gives you your global access stuff like "mqtt client" and "clock".  A minimum deployable service has more lines in it's deployment descriptor (5) than it does code, 2.

Code: [Select]

from home_auto_lib.mqtt import client
client.blocking_run()

The overall architecture is two MQTT buses.  I call them "Prod" and "Internal".  "Prod" is where any random device connects to the network.  "Internal" is the service bus.  On this bus all message formats and topics are normalised into a "data fabric".  The concept of a "proxy" servers as integration between these two buses.  To be accepted onto the internal service bus the translations usually happen in the proxy components.  This is probably the hottest area for work when new devices arrive in the flock.  It needs a good dose of engineering, it's a mess to be honest.

The first thing that jumps out at me when I go to HA is that it's a single process, monolithic python application.  It is also a state machine, in memory store.  Any and all  "in flight" transactions are in memory.  If you switch it off and on again it drops everything in flight.   It attempts to handle "everyone's" problems.  It manages to solve 90% of 90% of peoples problems.  If you buy into the eco-system and want something in those other 10%s...  Good luck.  My own "feeling" is that the added complexities introduced with the intent of solving someone else's problem actually hinder development on the more bespoke and lower levels.

It's not "wrong", it's just that by the time HA and others came along I had already seen the problems I will face in the future and decided to side step them before I wrote any much more code.  Instead of managing the growing complexity in a big monolithic "state machine", I split everything up across the network such that no code "could" "borrow" or modify state from another.  Making all processes expose their guts onto the message bus and making the message bu the source of truth, REALLY simplifies the half of it.

It is entirely routine for me to restart services rapidly while making dev changes.  The system doesn't care.  Very often if a bug does persist and a service fails, all the others just carry on.  The specific functionality provided by that service stops though.  Usually I notice as a gap on a graph in Grafana.  If it's late, I just "restart" the whole stack.  Figure it out later.  This relies on the project rules of engagement that all services should be stateless and emphemeral.  Shooting them in the head and a new one will spawn in it's place and take over unbothered.... unless it continually crashes and restarts.

I had a bug with a heating service where it took it's inputs, make it's choice and published it's requests.  It then hit a typo in a logging statement and crashed the service completely to console.  It got restarted.  The heating continued to function absolutely 100% without lose of functionality for 2 months last winter until I fixed it.  The poor services was crashing about 2 dozen times a day.

It's not perfect, but is it is full control.  The libraries were not developed as an entity in its own right.  It is code that was already peppering services, so they migrated into a "common" and the services could be split up into several packages, there are a few dozen scripts and another dozen or so support files.   This is my complete bespoke setup.  The library and core components are about 6 files.

There is no persistence internal to the service bus.  The "expectation of volatility" rules of engagement just ... side step this entirely.  Don't persist.  Rebuild.  There are caveats of course.  For persistence, you probably guessed it.  I use services.  The only service that exists that qualifies as a persistance "write" service is the "Influx Mqtt Bridge" which takes bus data and publishes it to InfluxDB.  The only persistence "read" service is the "Manager" component which is responsible for generating and publishing the master clock pulse and the static config data.

In fairness, if you want most of the system to simply "stop".  Stopping that clock service will suspend most of it.

EDIT:
A funny thing we do in software design, sometimes, is to make components of the system "actors" and describe their role as if it was a person describing their role.

I as a "Paul's Home Auto" service will subscribe to data changes on the following topics of interest.....
If I need config I shall subscribe to the topics that contain it and response to updates.
I will patiently wait for that data and not act without it.
If I have incomplete or invalid data I should always default to "safe".
If I cannot do the "right" thing, I shall do the "safe" thing.
If I wish to know what time it is, I shall ask the master clock, I shall NEVER ask the OS.
When I have my data I shall make my determination and notify other services by publishing my desires on the bus.
If I fail...  I shall die to be respawned a new.
Where possible, I should "Keep calm, log it, and carry on."

[abeyer]

Working software and money.  It is only when both of those are satisfied and delivered can be talk about bonuses like making friends.

And neither of them tend to happen alone in large projects, but by working as part of a team. And they both become harder to achieve at all if your team ignores/circumvents/sabotages you because you're perceived as unfriendly. "Making friends" isn't the goal, but having friendly professional relationships with the rest of your team is firmly on the happy path to get there. At least the way you're responding in the link, you seem to be leaning strongly into the opposite.

[paulca]

Working software and money.  It is only when both of those are satisfied and delivered can be talk about bonuses like making friends.

And neither of them tend to happen alone in large projects, but by working as part of a team. And they both become harder to achieve at all if your team ignores/circumvents/sabotages you because you're perceived as unfriendly. "Making friends" isn't the goal, but having friendly professional relationships with the rest of your team is firmly on the happy path to get there. At least the way you're responding in the link, you seem to be leaning strongly into the opposite.

Crossed on the wire, but after a lot of conversations escalating things up stream... Having found some support. 

I finally remembered my priority.  It's not that the project, it's documentation and processes make sense to "ME".  It's VERY important it make sense to those more junior to me.

I should not be isolating myself.  I should be becoming their leader.

I can empower myself easily by supporting others through the turbulence as I have the experience and the knowledge to guide.

Management can challenge me.  Of course they can.  But I am a software engineer.  I engineer software.  That is a 90% people problem.  You have deliverables, you have resources.... "people".   You typically (as an engineer) have no control over requirements or the people you are given.

The way I have previously embraced this and have forgotten, is to ... defend the team.  Lead from within.  Feel and express the pain the team feel.  Become a "gate" for BS.  Defend the team.  Defend the progress. 

I don't care who stands in my way, if they stand between me and "working software which meets the customers needs" they stand apposed.

My realisation is that, this is exactly what any and all juniors and seniors WANT to do.  If you empower them they will flourish.  Suffering because they have no voice to speak out.  I will be their voice.  It's easy to shoot down a 2 year junior.  It is not so easy with me.

I have done this before.  I know how.  It's a change of protocol.  Focus on the people.  Without the people you have nothing, you will achieve nothing.  Support the people and they will follow you.  If I can "empower" and motivate even two people I can double output.

[abeyer]

If you empower them they will flourish.  Suffering because they have no voice to speak out.  I will be their voice.

...said Spartacus, shortly before being crucified.

[The Soulman]

Paulca, I believe you are on the right track.

A while ago I was reminded here by someone here (can't find it anymore) that all humans are basically black-boxes with an data input and a data output.
With no or poor input data most will produce bad or useless output data, or simply near random noise that is hard to filter.
But with the correct input ALL black-boxes can produce good output data.
Key is finding the required input format, data-size and timing, it is different for each box but a solvable problem. 

[Siwastaja]

Extremely opposite viewpoint: programmers who require carefully crafted exact inputs to produce useful results can be replaced by AI, and said people are quite afraid of losing their jobs, some actually are already. We would need more programmers who are self-managing, have an idea of what is The Right Thing To Do (regarding user experience, software architecture, etc.), big picture and so on.

Then again, I also realize this process does not scale. If you are doing something complex and large, you can't find such geniuses and even if you could, they would disagree with each other and are hard to synchronize. So we need accurate descriptions, ticket systems, and hierarchies of leadership and planning.

The first extreme is useful in small to mid size projects with right people, but dysfunctional otherwise. The latter is dysfunctional in small projects as it leads to overengineering. Only use the bureaucracy when it's truly needed.

The trick is to find the balance between these two extremes.