Project isolation, social islanding?

[paulca]

So, for example sake, you are an experienced, qualified electronics person and you have a project you have been working on for decades.  Not a project you haven't finished, but a project that you just keep refining and evolving.

Let's say for the strawman it's making hardware devices for automation.  Digital actuators of many forms.  It's only a strawman for example structure.

After decades of development, you find that there are competing "packaged" off the shelf alternatives to many bits of your system, if a little imprecise and inflexible.

You occasionally run into a deep problem that takes a lot of thinking and you write out an elaborate article style topic on the subject and publish online... you get back.

"Why not use, <insert rubbish software here>"
"Why not use <insert AliExpress module here>"
"Why do you make it so complicated?  Just put a relay on the + of the motor."
"Why do you use transistor to separate circuits at all?  Just buy this module on Amazon and completely redesign your stuff for less functionality and more cost"
"I do it with 3 lines of firmware and it's fine for me."
"I think you miss-understand how heating control works.  If you look at GoogleNest....."

^^ This is exactly where I am in home automation software/firmware as a software engineer.  Nobody understand me anymore, the 99% majority use HomeAssistant and if something can't be done the HomeAssitant way it can't be done...  you are clearly wrong and over complicating your requirements if HomeAssistant doesn't already do it by default.  You must be an idiot.  Just assimilate the AliExpress modules and HomeAssistant least thy blasphem!

Does this analogy ring true for any of you guys since the birth of Arduino and the likes?

I have found that trying to explain the only reason they see things as "simple" is because they went the "happy path" and never considered ANY of the reality corner cases.  They even confess to stuff consistently breaking.  But, yet, I over complicate things.

That or they wrap a complicated concept I was trying to explain the how and whys of, but "Dude, you over complicate the WidgetX from <insert smart home garage cloud thingie>", without even the slightest realisation that... someone... somewhere already designed that one.... for you.

Anyone?  What is the solution?  Just shut up and carry on?  Try and find a better audience?  Abandon my superior bespoke system for a bloated (equally) opinionated "one size fits all" limiting eco-system?

[SteveThackery]

What an interesting topic!

I bet there are loads of stories like yours. There's a lot to be said for standards, even de facto standards like Home Assistant, and for many people the shortcomings are outweighed by the widespread and multi-sourced nature of the standard product.  As for functionality, quite often "good enough" really is good enough.

There is an element of risk in going for a bespoke solution, or the "Rolls Royce" solution like yours, because it is single-sourced and therefore might become unsupported for a number of reasons.

I'm looking forward to hearing other points of view - it'll be an interesting thread.

[ebastler]

Those casual readers of your post are clearly giving the wrong advice; they have not even understood the specific technical challenge you are facing and the full scope of what you want to achieve. But your response clearly can't be "Hey you guys, shut up and listen closely while I explain the intricacies of my design to you!". They simply won't be interested -- it's not their problem; their problems are simpler; and there are much simpler solutions to their problems.

So yes, you could say that you need to "find a better audience". Which audience may or may not exist, depending on how deep the rabbithole is which you have burrowed into.

If you should indeed find yourself in a situation where you can't find any kindred spirit interested in your system, maybe that is a reason to declare that project "finished" and turn to something new: While there is a lot of satisfaction in designing something very elegant and "perfect" and seeing it work, there also is the rewarding experience of joint problem solving, showing and explaining a good solution to others, learning from others etc. When that aspect goes missing because your project becomes too specialised/complex/esoteric, you are missing a significant part.

On the other hand, if you enjoy the challenge of perfecting your own project, and don't mind the fact that it can be more time-consuming than adapting something off-the-shelf, and are ready to work though the occasional frustrating phases of "lonely" problem-solving -- by all means go for it! It's a hobby, so the value is not measured in productivity.

To stick with your example: "I'll cobble something together in HomeAssistant because that will get me a working (though ugly) solution quicker" is not a good reason to abandon your own project in my opinion. "I'll consider my current home automation good enough and focus on a new hobby project, where the exchange with others is more rewarding" may be a good reason, depending on whether that aspect of the hobby is important to you.

[coppercone2]

this is why big companies spend an absurd amount of money on marketing R&D to be able to give accurate and truthful answers when comparing products.

The more basic it is, and the more generic your audience is, the more effort it takes to convince them... but its basically like a department in a company responsible for this.. and even they get effected by trends, there could be a down turn because of a hot product (that takes a while to get a generally bad reputation). They develop a marketing method, evidence, and reputation and then actively use those things to maintain their user base and have to constantly gather data to maintain relevance and legitimacy.

Its like bouer vs milwakee tools. milwakee has a hell of a time convincing the public why their tools are a better investment, but with tools its easier. Home goods... that's mass market stuff.. its a whole different level

If a major retailer adopts a new low end product from a existing market segment (i.e. something they sell in home depot) your in for some chop



More often, there is disruptive products that initially offer high quality, and then decrease in quality (i.e. amazonbasics batteries). In short, for a period of time someone is basically highly undercutting you (a big guy selling high end stuff for cheap) to build a customer base, then eventually they decrease quality to gain profit. Eventually word gets out, but its disruptive in the industry. in the case of amazonbasics, IIRC they were reselling very high quality cells made by panasonic, and did a swaparoo eventually.... but reputation takes a while to fall...

And on the other side of things (rare), sometimes someone does make a better product that is cheaper and then something goes extinct.


The big companies are almost.. like epidemiologists... when they fight some viral trend. They are very vigilant about sales data and respond quick.

[paulca]

But your response clearly can't be "Hey you guys, shut up and listen closely while I explain the intricacies of my design to you!". They simply won't be interested -- it's not their problem; their problems are simpler; and there are much simpler solutions to their problems.

Agreed.  Found this several times to the determent of my "community karma" scores in various online communities.

As my social and "corporate game play mentor/peer" in work says, "There is a difference between being right and communicating it successfully.  The later is always harder.  The more technical and detailed the harder it will be to justify it to a wider audience".

[nctnico]

I have found that trying to explain the only reason they see things as "simple" is because they went the "happy path" and never considered ANY of the reality corner cases.  They even confess to stuff consistently breaking.  But, yet, I over complicate things.

My observation is that lots of people are OK with products not working 100% and/or failing prematurely nowadays. To stay on topic of home automation: take Wifi connected doorbell-cameras for example. People with bad intentions will jam the Wifi before entering so the camera doesn't work when you need it the most.

[tszaboo]

^^ This is exactly where I am in home automation software/firmware as a software engineer.  Nobody understand me anymore, the 99% majority use HomeAssistant and if something can't be done the HomeAssitant way it can't be done...  you are clearly wrong and over complicating your requirements if HomeAssistant doesn't already do it by default.  You must be an idiot.  Just assimilate the AliExpress modules and HomeAssistant least thy blasphem!

Yes. With the Arduinos and easy to cobble together modules, electronics or home assistant reach wider audiences.
Which means that more people with very shallow knowledge on the topic will talk about it. One of the reasons it's important to define what sort of solution you are looking for. And ignore some of the answers.

[paulca]

^^ This is exactly where I am in home automation software/firmware as a software engineer.  Nobody understand me anymore, the 99% majority use HomeAssistant and if something can't be done the HomeAssitant way it can't be done...  you are clearly wrong and over complicating your requirements if HomeAssistant doesn't already do it by default.  You must be an idiot.  Just assimilate the AliExpress modules and HomeAssistant least thy blasphem!

Yes. With the Arduinos and easy to cobble together modules, electronics or home assistant reach wider audiences.
Which means that more people with very shallow knowledge on the topic will talk about it. One of the reasons it's important to define what sort of solution you are looking for. And ignore some of the answers.

Yes.  It's more the devastating social consequences of doing so in most communities.

There are people in this thread whom I have spend weeks in debate with on similar topics in firmware and I came away the winner as they eventually got through to me there WAS a VALID and simpler alternative.  I accepted this, embraced it, thanked him for it and moved on with it.

I wish for the same in home automation.  I want to be told my solution is incomplete or invalid, but I need to be told why. 

For this to occur I need to communicate the problems I am trying to solve.  This is where the problem comes on "wider audiences", especially, or more so on the likes of reddit, than to here.

At the same time, "here" is electronics and while there is tolerance for topics like software the audience is limited.

[paulca]

I felt it necessary to highlight....  that 99% of my system "integrates" via an open message bus.  100% (or close as I can get to it) of state is on the bus.

This means that ALL code variables come from the MQTT bus.  That means, the entire internals are exposed for any and all third party or second party integrations. 

In other words, it can work with or without home assistant.  Home assistant can work with it.

It's actually so well integrated with the likes of home assistant if I install home assistant locally and get it to scan for things it comes back with 100s of things.  It has found my service bus topics.  It has found the state of my application.  It like any other message client is 100% free without restrictions to publish to those same topics.  If the messages make sense and are valid they will be actioned.

It's not like I am proposing people stop using Home Assistant and use "My emperor's new clothing range".  It's not like that at all.

[exmadscientist]

Unfortunately I think this is a very general problem these days, and it is only getting worse. Most people who want to talk about a topic do not really understand it. People with deep understanding, who can listen to you and what your goals are, and then are able to explain things in any kind of useful way... those people are worth their weight in gold. Interestingly, I think the rarest part, or just least obvious skill, is just finding people who can listen well and understand you rather than thinking you said whatever they thought. Listening really is the most powerful "soft" skill!

(And yes, I'd say there are a few such people around here.)

[paulca]

(And yes, I'd say there are a few such people around here.)

We are all guilty at times.  I know I do it. Often however I am presenting my straw man in the hope that someone shows me how easily burnt it is.

When people come at your "iron man", something you have re-inforced for thousands of hours with matches and sticks and the general wider concensous supports them... it hurts.  No, it leads to your questioning you communications skills, or worse, your entire way of thinking, working and solving problems.

At least this thread is on the supportive side.  Thanks.  Feel free to throw a few I can take them, my ideas are often radically and often radically wrong.  I like to endure conversations and debates, in detail as to the "But why".

I think these types of debates and all manor of opinions and levels of pendanty are handled on this forum partucularly well.  Disagreements happen.  Things get heated, things go tagent, but the discussion is open.  Ideas are only put down if they are stupid.  People that propose them are not.  This affords a "safe space" where ideas can be Staw-manned and set ablaze in public without calling anyone an idiot or like wise for proposing the idea.

It is better to generate 100 idea a day and find a gem once a month than to hinder yourself for generating ideas at all for fear of failure.

(too much beer.  I'm out for how)

[paulca]

(And yes, I'd say there are a few such people around here.)

I would also like to point out.  As a credit to this forum and some of it's members, there are those who go out of their way to help and not judge without experience.

There are also those who are strong enough, knowledgable enough to "forcefully" and "authoriativeily" say, "No, don't do that, that's stupid.  Do this instead."

Before ChatGPT revealed it's true colours and was "hobbled" there was a thread on here where someone asked ChatGPT about the forum members and it was able to speak very positively about some of the most frequent here.

It's a shame it got "nerfed" (ecause it could dox or dox-ography nearly anyone).  It was a very interesting thread asking Chat GPT for it's analysis on forum members and their contribution, helpfulness and like-ability.

[Geoff-AU]

As someone who has inherited projects worked on for decades, sometimes there is very much a better way to do things if you started doing it now.  But there are also baked-in lessons that would all have to be re-learned if you ripped it up and started again.

But also as someone who tinkers with hobby things until they're just the way I like, the value is primarily realised in my own satisfaction of it working the way I want it to work.  There's little additional satisfaction derived from telling people about it, because it's a solution that's unique to me.

I have a "dumb house" philosophy - I don't want to be updating firmware for light switches or other "Internet of Shit" devices.  If I can't turn things on and off without an internet connection I will scream.  A light switch lasts for about 30 years, you replace it, and then it lasts another 30 years.  The value-add of a phone app to me is zero.  But I will do smart "non-essential" things like monitoring energy flows and temperatures.  I have 3 temperature sensors on my hot water tank, because I can.  Top, middle, and bottom.  The middle sensor is the most interesting one because it indicates how much "reserve" hot water we have and shows the mixing behaviour of the tank during a heating cycle.

My friends have HA, and they love it, but they also know that I am a weirdo and like to do my own thing so they don't try and persuade me to change.

[Siwastaja]

Interesting discussion, I have encountered this behavior many times and at first it was confusing, but then I realized these who comment have rarely actually done anything. They are asking those questions because they don't know! The problem is that you are reading between the lines and assuming their questions are loaded - and maybe they really are, but try parsing their question literally and the whole game turns around:

"Why not just use Aliexpress module and Arduino to do this and that?"

You read it as: "This person thinks the task can be achieved by this Aliexpress module and is pushing that view, what an idiot"
Try reading it as an honest question: they are asking why not use an Aliexpress module because they do not know.

So try answering their question:
"Building this system from Aliexpress modules requires countless of hours of integration work and programming, and then maintaining."
"Availability of these modules is iffy. Even if you designed around something popular like Raspberry Pi and need to order 10000pcs, you may have to wait for a year."
"Specifications of Aliexpress modules can change at any time without a warning."
etc.

In context of what I have been doing in past and am currently doing, this is always the same case with stuff like ROS (Robot Operating System), or HomeAssistant - they are tools, not ready-to-ship products, yet some people seem to hope they are the latter kind. They don't know. They can be told about it, many believe, some don't, some people enter some kind of locked-up state and fail to communicate further.

We manufacture and sell a kind of home automation (energy management) box. It's absolutely and totally integrated solution, one full custom designed box with one full custom closed firmware, and one full custom server infrastructure. Installers wire it up and customers use it. No complex configuration, no programming work, no wiring up different modules. This is the only way to make something happen in a large scale. This is something >10% of homeowners could buy.

There have been some electricians who play around with HomeAssistant and Shelly relay modules etc. but this is like 0.01% stuff, total niche. It's not scalable unless there is another company who abstracts away that work, but even those have been so far unsuccessful: even though the ecosystem has been made simple, the fact you need to buy 5-10 different types of modules, wire them up, configure, buy an optimization cloud service from 3rd party is a total showstopper for 99.99% of sales companies and 99% of electricians. And only 0.01% of homeowners are A) electronics hobbyists B) have time and energy to put into their home automation.

Designing a product and/or service which can be sold to masses is a lot of work, and many of the design goals are completely different from a hobby project, or an engineering/science-driven R&D project. Decision makers (customers, installers, sales companies) hate complexity, they want to deal with a single company which can provide everything and have a phone number where to call for support.

[RJSV]

   First thing to come to mind;  My (private) rooms lacked capacity to sustain the hardware aspects of a large project.  That was after a bit of a 'crash' in resources.

   Somewhat in isolation, particularly due to IP concerns (intellectual property, on innovation).   Not so much on the technology, building mechanical elements for a completely mechanical computer (firstly for teaching and STEM type student introductions).
Rather, than IP on the devices, my concern was keeping that business model a bit quiet.   The IP stuff actually had an emphasis on covering many 'bases' as they say...to prevent some other larger entity from getting their own Patent(s) on associated devices / methods.

In other words, I threw in 'everything but kitchen sink' in my own Patent (applied for) application.   Not for eventually IP protection, but simply to clear a setting where I could operate and grow, without some other, big, entity interference.

   So, I've ended up here, 2024, with issues more along the lines of isolation (an 'isolated project's, like OP stated).   Hard to justify years and years of R&D progress, in a vacume with lack of real market involvement.
   I figure, can't really SELL the hardware with any significant quantity market;  maybe best to start an education oriented 'Training' system, as the business profit, while the mechanical computer itself being a tool, along with publishing various manuals and training aids

   Not your usual 'IC or transistor' manufacture, for sure.

   I'm reading your thoughts on how you (OP) present, or attempt to present ideas to others, as you likely need the support.   So, it seems a dilemma between needing isolation, meanwhile also needing support.
Plus, in my case, the teaching system project is heavily oriented to things old, now;   The 8-bit processor types, of 1970's, and engineering culture, of 1980's...that's now 40 years + !

   Can you (OP) find someone, a partner or investor, that fills in the skills you maybe lack...(A 'spokesperson' or marketing type that can spend the time, to hear your concerns and, uh, TRANSLATE, to a more compact expression.


   I've encountered others, friends and family, who do the kinds of work that I had anticipated doing, (as a STEM or technical beginners training organization) maybe sometimes just in the similar spirit, which has lots of 'inspiration' or engagement....a difficult item for teachers, often.

[paulca]

On brevity and compaction.  I remember Peter North (I think) on here in the MCU sub having the same issue.  If you omit all the details that you are aware of people will either ask about them or assume them.  So you need to clarify.   If you DO give them all the details it's tldr, wall of text, nobody cares.

It's a bit like that fine art of writing a good StackoverFlow question.  It's not even funny that there are Stackoverflow questions on how to ask stackoverflow questions.

[Just_another_Dave]

I think one of the reasons why you get so many replies recommending things like Arduino or home assistant is that those technologies tend to limit amateurs from gaining more knowledge about electronics. For example, Arduino is quite easy to use for newbies, but it limits significantly the features accessible to the user (many boards don't expose interruptions in Arduino ide). As a result, a broader separation between amateurs and professionals exists nowadays and in order to grow in this field, you need to change the tools and methods you use to others that are less amateur friendly

[RJSV]

   I have to admit, having some difficulty grasping the OP's question(s), involving 'social isolation'.   Maybe a bit more explanation what that meant (in post title).

   Reading posts, and replies, it appears like one single individual (OP) developing product(s), up against some large organized development and marketing entities.   I was originally interested, as, in my case, it's just the one engineer, contemplating interfacing with some large group (eventually).   Could you please address some of that, 'isolation'....mentioned due to need to keep your product secret, initially ?

   I have a lot of 'pet peeve' with portions of home automation, that being the video media remote controls, and the layers of complexity being introduced (...pushed), by market.   Engineering is one, mostly logical thing.   But once the products get into marketer's hands, the agenda swiftly shifts (in my view), virtually IGNORING customer desires, and 'responding' in nearly 100% manipulative ways...
In other words, customer complaints usually imply that they get second priority, and the marketers confirm that, by the response (to complaint issues).
   I cannot necessarily condem the marketers view, as they eventually 'pay the bills'.

[abeyer]

On brevity and compaction.  I remember Peter North (I think) on here in the MCU sub having the same issue.  If you omit all the details that you are aware of people will either ask about them or assume them.  So you need to clarify.   If you DO give them all the details it's tldr, wall of text, nobody cares.

It's a bit like that fine art of writing a good StackoverFlow question.  It's not even funny that there are Stackoverflow questions on how to ask stackoverflow questions.

I feel like some of this is the limitations of the forum format in general. Something like a wiki, or github repo, or anything that prescribes a more structured format while still having some form of feedback/discussion can make this more approachable, in my mind. Stackexchange is a step towards that... particularly if you use the community wiki pages and similar, but is maybe too much focused on the q&a format vs something more project focused.

[vk6zgo]

So, for example sake, you are an experienced, qualified electronics person and you have a project you have been working on for decades.  Not a project you haven't finished, but a project that you just keep refining and evolving.

Let's say for the strawman it's making hardware devices for automation.  Digital actuators of many forms.  It's only a strawman for example structure.

After decades of development, you find that there are competing "packaged" off the shelf alternatives to many bits of your system, if a little imprecise and inflexible.

You occasionally run into a deep problem that takes a lot of thinking and you write out an elaborate article style topic on the subject and publish online... you get back.

"Why not use, <insert rubbish software here>"
"Why not use <insert AliExpress module here>"
"Why do you make it so complicated?  Just put a relay on the + of the motor."
"Why do you use transistor to separate circuits at all?  Just buy this module on Amazon and completely redesign your stuff for less functionality and more cost"
"I do it with 3 lines of firmware and it's fine for me."
"I think you miss-understand how heating control works.  If you look at GoogleNest....."

^^ This is exactly where I am in home automation software/firmware as a software engineer.  Nobody understand me anymore, the 99% majority use HomeAssistant and if something can't be done the HomeAssitant way it can't be done...  you are clearly wrong and over complicating your requirements if HomeAssistant doesn't already do it by default.  You must be an idiot.  Just assimilate the AliExpress modules and HomeAssistant least thy blasphem!

Does this analogy ring true for any of you guys since the birth of Arduino and the likes?

I have found that trying to explain the only reason they see things as "simple" is because they went the "happy path" and never considered ANY of the reality corner cases.  They even confess to stuff consistently breaking.  But, yet, I over complicate things.

That or they wrap a complicated concept I was trying to explain the how and whys of, but "Dude, you over complicate the WidgetX from <insert smart home garage cloud thingie>", without even the slightest realisation that... someone... somewhere already designed that one.... for you.

Anyone?  What is the solution?  Just shut up and carry on?  Try and find a better audience?  Abandon my superior bespoke system for a bloated (equally) opinionated "one size fits all" limiting eco-system?

Many years ago, the commandment came down from Mt Olympus that we had to convert all the attended TV & Radio Broadcast Transmitter sites to Unattended operation, with all the conversions done "in-house" by the existing staff.

We did receive a "suggested" plan, which looked like it had been "slapped together" by someone at the office between Morning Tea & Lunch, which was totally inadequate for the job.

At my workplace, initially, in order to meet the ridiculously short timeline, we had a horrible "bodge" in place, which sort of did the job, then we got a few "bits n' pieces" supplied for the final iteration.
The core of this was a "Programmable Logic Controller", which received the inputs from various sensors in the form of relay closures, did the "smart stuff" then operated the correct interface control via a relay closure.

One bloke who had played with PLCs before got the job of programming the thing, using the little keyboard that came with it, whilst the rest of us tried to sort out the multitude of incompatible interfaces of the transmitters, programme switchers, alarms, etc.

One of the first obstacles was that the multi-cored cables supplied with the "conversion kit" were not rated for the control voltages of the (then) 28 year old Vision & Sound transmitters which were 65v DC & AC respectively.
The control functions were already available at the control desk, so the easiest way was to use a relay panel to solve the voltage rating problems, plus provide additional relay contacts that we found we needed.

As we went through things, we constantly encountered functions that the "auto control" needed that hadn't been anticipated, so when the PLC programming bloke would come out with the latest version of the program & run through it, inevitably, someone would say "what if?"
He would exclaim "Arrr, Shit!!!" & go away "Tae think again", like unto "Proud Edward" in the song.


For my sins, I was the "Project Leader" & the whole thing was horrifically stressful.
I developed a "facial tic", which I always had thought was something fanciful thought up for Inspector Clouseau's superior in "the Pink Panther".

[John B]

Not really sure about the specifics of what your setup is, but I gather you don't utilise any of the large automation engines, whether home focused like Home Assistant or OpenHAB, or something generalised like Node Red?

I've really enjoyed getting into OpenHAB despite it not being the top dog in (open sourced DIY) home automation.

Home automation has attracted people who enjoy tech in their houses, without understanding much about underlying technology or it's pitfalls.

Even Home Assistant is very small in comparison to controlled proprietary setups from Google or Amazon.

Home Assistant has broadened it's appeal to those wanting to getting a bit more into the DIY aspects, while still having the luxury of equipment that can be connected by installing an addon. While there are many very competent people using HA to an impressive extent, having a large community also opens the floor to more noise.

[JPortici]

Does this analogy ring true for any of you guys since the birth of Arduino and the likes?

Yes. Trying to make someone understand why arduino or platformio is crap, or why they have failed as educational tools because they have never tried to create a product with them is pointless... because they are not making products, but contraptions.
(yes yes yes they helped the masses approach the field but god forbid they try and create something, it's always follow this  tutorial and use that library, never create something on your own because they took the shortcut and have no real knowledge of what they are doing)

[Smokey]

Does this analogy ring true for any of you guys since the birth of Arduino and the likes?

Yes. Trying to make someone understand why arduino or platformio is crap, or why they have failed as educational tools because they have never tried to create a product with them is pointless... because they are not making products, but contraptions.
(yes yes yes they helped the masses approach the field but god forbid they try and create something, it's always follow this  tutorial and use that library, never create something on your own because they took the shortcut and have no real knowledge of what they are doing)

what he said.

[Jeroen3]

Anyone?  What is the solution?  Just shut up and carry on?  Try and find a better audience?  Abandon my superior bespoke system for a bloated (equally) opinionated "one size fits all" limiting eco-system?

Yes. Just smile and wave when they go to reset and pair some cheap module for the dozenth time into Tuya so that Xi can control their lights.
It gives them a feeling of accomplishment because "they did a thing!". Your thing a bit more initial effort, but probably has a much higher WAF and reliability.

[coppercone2]

I had nothing but bad luck trying to make complex systems with arduino in a corporate level because of maybe EMI causing issues. It is supposed to go together using daughter cards and stuff but eventually it starts to crash and when that happens all the time saved on design is wasted... even with the high end modules meant for that sort of thing.

It does not scale with 'throw more money at it to solve issues'.