How to merge many forks from github, 108 many?

[RoGeorge]

Official but neglected master project https://github.com/python-ivi/python-ivi
Main contributor fork https://github.com/alexforencich/python-ivi
All the 108 forks on github https://github.com/alexforencich/python-ivi/network/members

The project is a Python module, and so far it seems to be the most complete implementation of the IVI standard for controlling measuring instruments, open and multiplatform.  I'm interested mostly in those forks that include Rigol instruments because this is the only brand of SCPI instruments I have so far.

Is it reasonable to try to merge all the 108 forks (many don't have the Rigol instruments I need)?
Would merging all that be a one hour task, a one month, or rather impossible?

What to do to get a reasonably updated version?

[alm]

I haven't looked at the forks recently, but anything that adds or improves a single instrument driver would be fairly straightforward to merge, while anything making more sweeping changes would be difficult. So all 180 would probably be days of work even for an experienced developer, and possibly more, but if you do a limited subset you might be able to do something in a couple of hours if you've done this before and are comfortable with the git command line.

I was at one point considering offering to help maintain and at least merge PRs that are complete with tests etc, since I wrote a number of instrument drivers myself. But I don't have the time right now.

[RoGeorge]

Thanks for the advice.  Now I remember we talked about the same abandoned proj, this python-ivi, about two years ago:
https://www.eevblog.com/forum/metrology/software-for-automated-testing-(scpi)/msg3398008/#msg3398008
and I might still have the tools I used back then to identify the forks that have Rigol drivers.

Back then only 10 forks were having drivers for Rigol.
Will post in the old thread in case I'll try to merge the relevant forks again.

[DiTBho]

Which wave gen and dso do you need? (Rigol parts)
It should be cleaner to write dedicated drivers.
Sigrock should have something.

[RoGeorge]

I have a DSO from the DS1000Z series, DS1104Z, an AWG from the DG4000 series, DG4202, and a power supply from the DP800 series, DP832A, all 3 from Rigol.

The ones I was needing were the IVI drivers (python-IVI drivers more precisely) for the AWG and for the DSO.  Such an IVI driver is suppose to translate an IVI command to lower SCPI commands specific to each instrument.

I'm not sure Sigrok is using Python-IVI at all, I think it doesn't use IVI commands at all, there are other kind of drivers.

In IVI, the word driver is somehow misleading, it's not like a mouse driver which is OS specific.  IVI drivers are, briefly said, like a dictionary between a generic set of IVI commands (e.g. all instruments of type "Oscilloscope" will implement the same sets of IVI commands no mater what model the oscilloscope is, or who made it, while for SCPI commands each model uses its own set of SCPI commands that are not compatible between models or between manufacturers) and the specific SCPI corresponding commands of a particular model of instrument.

Unfortunately, the IVI standard was defined for Microsoft's C# or .Net, and the python-IVI implementation will never be 100% compatible because of some language specific properties that are different between Python and .Net or C#.  That is why, the IVI drivers offered by the instrument's manufacturer are useless for the Python-IVI implementation of the IVI protocol.

Might be interesting to try to run the manufacturer's (Microsoft) IVI drivers (in Linux) using WineHQ, or mono, but even if that would work, it looks more like a stunt, will have yet another proprietary binary blob that might not even work at a future time.

So far I've temporary parked the forks merging, and spent all my time with the other topic, https://www.eevblog.com/forum/programming/extract-precise-amplitude-and-phase-from-a-frequency-sweep-(vna-from-dsoawg)/

[Sherlock Holmes]

Official but neglected master project https://github.com/python-ivi/python-ivi
Main contributor fork https://github.com/alexforencich/python-ivi
All the 108 forks on github https://github.com/alexforencich/python-ivi/network/members

The project is a Python module, and so far it seems to be the most complete implementation of the IVI standard for controlling measuring instruments, open and multiplatform.  I'm interested mostly in those forks that include Rigol instruments because this is the only brand of SCPI instruments I have so far.

Is it reasonable to try to merge all the 108 forks (many don't have the Rigol instruments I need)?
Would merging all that be a one hour task, a one month, or rather impossible?

What to do to get a reasonably updated version?

Just to clarify, you don't merge forks in Github, you merge branches. A branch might well be in a fork, but all merging is at branch level.

[Sherlock Holmes]

Official but neglected master project https://github.com/python-ivi/python-ivi
Main contributor fork https://github.com/alexforencich/python-ivi
All the 108 forks on github https://github.com/alexforencich/python-ivi/network/members

The project is a Python module, and so far it seems to be the most complete implementation of the IVI standard for controlling measuring instruments, open and multiplatform.  I'm interested mostly in those forks that include Rigol instruments because this is the only brand of SCPI instruments I have so far.

Is it reasonable to try to merge all the 108 forks (many don't have the Rigol instruments I need)?
Would merging all that be a one hour task, a one month, or rather impossible?

What to do to get a reasonably updated version?

There are a lot of forks and some of them now have multiple branches and some of the forks are forks of other forks.

I don't think Github lets us take a birds eye view of all branches and all forks, that would certainly help answer your question.

But here's an interesting factoid about Github, all commits, on every single fork, are all actually in the original repo. The forks are "virtual" repos so to speak, they are not distinct repos as is often assumed.

This isn't much help with your question though, I'll need to think about this, its a good question.

[Sherlock Holmes]

Frankly this is complex, to all intents and purposes you're looking at a large number of divergent and possibly incompatible variants of the original repo.

In addition to which you must factor in the fact that each fork was created at a different date and time and so each fork - could - be a fork of the original repo at a different state...

Also, although you have a fork and have the most recent activity of any fork, you do not have this commit (one example among many)

https://github.com/acconeer/python-ivi/commit/df7945df377b253955efa0c25e3f2c8f8f7eec4c

This is a serious problem with Github and open source, unless fork owners and repo owners stay on the ball, the code just slowly bifurcates and there is no "current" version or "best" version, ideally contributors would be creating PRs back to the main repo and the repo owner would be diligent enough to either merge or provide feedback, fork owners should also regularly (routinely) pull from the main repo too to pickup whatever has been merged from other contributors, but this is often never done.

[SiliconWizard]

Yeah. That's something I pointed out a while ago regarding how people use github in general and how inefficient it is when projects get split up across tens or hundreds of forks. Nobody will ever care or even have the time to consolidate any of that. It's just not doable for maintainers. For mere "users", you have to pick one of the forks that's closest to what you need and stick to it.

Even if you managed to properly merge 108 branches, which could take weeks, while not introducing horrible and intractable bugs, that would be a one-time affair. And now what? Would you keep doing that on regular basis watching modifications across 108 forks (which may each have several active branches?)

I have no secret sauce for that. I tend to only use open-source projects that have active official maintainers and stick to the official project. Too much of a mess otherwise. For the rest, I write my own stuff.

[Sherlock Holmes]

Yeah. That's something I pointed out a while ago regarding how people use github in general and how inefficient it is when projects get split up across tens or hundreds of forks. Nobody will ever care or even have the time to consolidate any of that. It's just not doable for maintainers. For mere "users", you have to pick one of the forks that's closest to what you need and stick to it.

Even if you managed to properly merge 108 branches, which could take weeks, while not introducing horrible and intractable bugs, that would be a one-time affair. And now what? Would you keep doing that on regular basis watching modifications across 108 forks (which may each have several active branches?)

I have no secret sauce for that. I tend to only use open-source projects that have active official maintainers and stick to the official project. Too much of a mess otherwise. For the rest, I write my own stuff.

It's kind of a shame, there are probably umpteen very good disparate changes scattered across some of these forks. alas they will remain forever unknown!

[RoGeorge]

I don't think Github lets us take a birds eye view of all branches and all forks, that would certainly help answer your question.

It has something, it's under the 'Insights' tab of each repo:
https://github.com/python-ivi/python-ivi/network/members
https://github.com/python-ivi/python-ivi/network

[alm]

But here's an interesting factoid about Github, all commits, on every single fork, are all actually in the original repo. The forks are "virtual" repos so to speak, they are not distinct repos as is often assumed.

Are you saying a Github fork is not like a Git clone, in creating a copy of the original repository that references the original, like what the documentation says? Do you have a reference for how this works on a Git level? This seems like an awful lot of effort for them to go through beyond the basic Git functionality just to safe some duplication. It would also imply all forks disappear when the original repository is deleted.

If I were them I'd just implement some object-level deduplication agnostic of any repository boundaries.

[ledtester]

But here's an interesting factoid about Github, all commits, on every single fork, are all actually in the original repo. The forks are "virtual" repos so to speak, they are not distinct repos as is often assumed.

Here is an example of this...

This fork:

https://github.com/jkc-sw/python-ivi

is 4 commits ahead of the original repo: https://github.com/python-ivi/python-ivi

and this is the latest commit on the jkc-sw repo:

https://github.com/jkc-sw/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1

Now replace the "jkc-sw" in the url with "python-ivi":

https://github.com/python-ivi/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1

and github will show you the same commit but also report:

! This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

This suggests that the jkc-sw and python-ivi repos are sharing the same physical git repo.

...
It would also imply all forks disappear when the original repository is deleted.
...

You would only delete the underlying git repo when all forks have been deleted - like a reference counting / garbage collection type of situation.

[Sherlock Holmes]

But here's an interesting factoid about Github, all commits, on every single fork, are all actually in the original repo. The forks are "virtual" repos so to speak, they are not distinct repos as is often assumed.

Are you saying a Github fork is not like a Git clone, in creating a copy of the original repository that references the original, like what the documentation says? Do you have a reference for how this works on a Git level? This seems like an awful lot of effort for them to go through beyond the basic Git functionality just to safe some duplication. It would also imply all forks disappear when the original repository is deleted.

If I were them I'd just implement some object-level deduplication agnostic of any repository boundaries.

Yes Sir, a GitHub fork (as you know, forks are not part if Git itself) is essentially a set of user specific branches in the main repo, there is only one repository, the repo is not in any sense copied when we fork it.

Have you noticed how fast it is to create a fork? even of a huge repo?

The evidence for this is found when one ues Octokit to access and pull repository metadata, if you do that you'll be able to see that commits that exist in a fork also exist in the main repository, they are present in that main repository.

I think that under the hood forks are just per-user branches where the user name is invisible. So "Janet" forking some repo that has a "master" and "version2" branch actually is just the creation of a "Janet.master" and "Janet.version2" branches and the system/UI etc simply hides the "Janet." part of the name.

In this sense forks are "virtual repos" nothing but a set of branches, physically copying repos strikes me as a terrible idea and one that could lead to hugely complex confusion if things ever went wrong, the "virtualizing" is absolutely the way to go.

[alm]

and this is the latest commit on the jkc-sw repo:

https://github.com/jkc-sw/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1

Now replace the "jkc-sw" in the url with "python-ivi":

https://github.com/python-ivi/python-ivi/commit/5f8356ebd39041673b5546d77bc202295550fbe1

and github will show you the same commit but also report:

! This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

This suggests that the jkc-sw and python-ivi repos are sharing the same physical git repo.

Interesting. This is little help when merging from the command line, however, as you would be doing when merging a hundred or so different repositories, because the commit is not included when cloning the python-ivi repository:

Code: [Select]

% git clone https://github.com/python-ivi/python-ivi.git
Cloning into 'python-ivi'...
remote: Enumerating objects: 4901, done.
remote: Total 4901 (delta 0), reused 0 (delta 0), pack-reused 4901
Receiving objects: 100% (4901/4901), 1.25 MiB | 3.84 MiB/s, done.
Resolving deltas: 100% (3985/3985), done.
% cd python-ivi
% git show 5f8356ebd39041673b5546d77bc202295550fbe1
fatal: bad object 5f8356ebd39041673b5546d77bc202295550fbe1

Only when cloning the jkc-sw repository will the 5f835 commit be included. So as far as Git is concerned they are two separate repositories with two different sets of commits.

[ledtester]

I think git fetch will do the job for you. See this SO discussion:

https://stackoverflow.com/a/2949855/866915

[RoGeorge]

For this particular merge I was interested mostly in those forks containing Rigol drivers, which drivers happen to be in a specific folder named rigol.

I've copy/pasted all the user names from the forks webpage, concatenated the list with mkdir and git clone, then cloned all the 108 forks locally.  Then made a search on the local disk, and eliminate those forks without the rigol directory.  Only 10 forks remained, which I diff with 'meld', looking only at the Rigol instruments files.  All 10 were about the same so very easy to merge, the most recent/active fork for the Rigol drivers was the one from 'nicedwarf'.

Then I've made a 109'th fork, then realized it's not possible any longer to submit changes from a local git back into github, because 'security' and extra hoops with security tag from Microsoft and so on, so I've said good-bye github, and uploaded my fork with its few changes to gitlab instead of github:  https://gitlab.com/RoGeorge/python-ivi  Gitlab is still sane, thank you.

But then I've realized it would be easier to not use any of the python-ivi, pyvisa, pyvisa-py, python-vxi11, instrument specific drivers and/or proprietary VISA shared libs at all, and send SCPI commands to SOCKET instead. 

[DiTBho]

send SCPI commands to SOCKET instead

so, is it the definitive approach?
you made things very confused!

[RoGeorge]

Other protocols have their own pros and cons, depending on the end user, and depending on the end application.

- SCPI is plain text, trivially simple to use, but implies taking care of details, like dealing with possible errors, or learning the exact SCPI commands known by the instrument, or setting the instrument address and so on.

- VISA will abstract the physical layer, and present the same software interface no matter how you want to talk with your instrument, either through LAN, USB, serial port or GPIB.  VISA has some extra features regarding the messages exchange, too, not only about abstracting the physical layer.  All that might lower the level of details needed to be known by the end user.  I only use LAN connectivity, with fixed IP addresses, don't need to discover my instruments in the LAN, and already know their capabilities.

- IVI will abstract the instrument model/manufacturer, and try to make all the oscilloscopes the same, all the generators the same, and so on.  IVI will also unify the various SCPI dialects into a unique set of IVI commands.  For that, it will need some sort of translator from IVI to SCPI instrument specific, so it will need an "IVI driver" (better name would have been "IVI translator"), which IVI driver is specific to a given model of instrument.  So apart from the hardware drivers (if needed), now an IVI driver is needed, too.  This IVI driver is specific to the IVI implementation, same as a hardware driver is specific to each OS.  You will need IVI drivers written specifically for your instrument AND for python-ivi.  For example, the COM-IVI drivers provided by the instrument's manufacturer are useless for python-ivi.


IVI and VISA might make sense in an industrial setup, where instruments can be configured by different operators, can be moved around from one lab to another, or can be replaced with newer models, and so on.  In a personal lab, where the instrument never changes, there is not much advantage of adding the extra layers.

I've tried IVI, and almost any setting is obfuscated by 2-3 layers of abstraction.  Had to dig through code sources way too often to learn how to change otherwise trivial settings.  Then, found myself sending IVI commands while sniffing my own TCP packets to know what SCPI commands were sent by IVI, and if they correspond or not with what I was trying to do. 

IVI and VISA are OK as industry standards, just not good for my personal use case.

Maybe I should write a step by step 101.  LXI-ready instruments are trivial to control remotely.  It all works out of the box with nothing but a LAN cable, no other software or drivers are needed.  Telnet to the instrument and type SCPI commands, or open a TCP socket from a programming language, and send SCPI from there.

[DiTBho]

I think one should focus only on the needed operations.
- frequency/div setup
- amplitude/div setup
- sample record
- sample dump

[SiliconWizard]

To sum up: write your own code and you'll have wasted a lot less time.

[alm]

To sum up: write your own code and you'll have wasted a lot less time.

That's always what it looks like at the beginning. Surely writing it from scratch will be faster and lead to a better result. Sometimes it's true, but it's equally common to after some weeks of work discover that reinventing the wheel is a lot more complicated than you thought, and the system you built might have ended up equally complicated as the complicated thing you tried to avoid.

[SiliconWizard]

To sum up: write your own code and you'll have wasted a lot less time.

That's always what it looks like at the beginning.

Well, I don't really agree with that. I would say almost the opposite: that's almost always what it looks like at the end.
At the beginning, code that appears to have been used by other people looks like integrating it to get what you want should be a breeze.

Surely writing it from scratch will be faster and lead to a better result. Sometimes it's true, but it's equally common to after some weeks of work discover that reinventing the wheel is a lot more complicated than you thought, and the system you built might have ended up equally complicated as the complicated thing you tried to avoid.

I'm not advocating reinventing the wheel when it doesn't make sense of course, but IME developers tend to shy away from writing their own code even for relatively basic functionality because they have been told it was bad not to reuse others' work. (And, sometimes - what I'm gonna say is politically incorrect - that's because they just are not capable of doing so.) So there you go, with all the bloat that follows and hours of debugging and managing versions of third-party stuff you have no control over. Especially when it's "random" online projects and all the worse if they are spread across hundreds of forks.

Now admittedly, and maybe that's what you had in mind as a counter-example, OTOH software developers have a tendency to want to write their own code for everything and discard others' code. So that's a constant battle between the two opposites.

[ledtester]

Here's an example of using git fetch to perform a merge of two repos. The commit "5f8356" isn't in python-ivi's tree but is in jkc-sw's tree so the "git show" command before the fetch fails but succeeds afterwards.

Code: [Select]

PS C:Temp> git clone https://github.com/python-ivi/python-ivi.git
Cloning into 'python-ivi'...
remote: Enumerating objects: 4901, done.
remote: Total 4901 (delta 0), reused 0 (delta 0), pack-reused 4901Receiving objects: 100% (4901/4901), 1.21 MiB | 1.91 MiB/s
Receiving objects: 100% (4901/4901), 1.25 MiB | 1.65 MiB/s, done.
Resolving deltas: 100% (3985/3985), done.

PS C:Temp> cd .\python-ivi\

PS C:Temp\python-ivi> git show  5f8356
fatal: ambiguous argument '5f8356': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

PS C:Temp\python-ivi> git fetch https://github.com/jkc-sw/python-ivi.git
remote: Enumerating objects: 20, done.
remote: Counting objects: 100% (12/12), done.
remote: Total 20 (delta 12), reused 12 (delta 12), pack-reused 8
Unpacking objects: 100% (20/20), 4.52 KiB | 53.00 KiB/s, done.
From https://github.com/jkc-sw/python-ivi
 * branch            HEAD       -> FETCH_HEAD

PS C:Temp\python-ivi> git show  5f8356
commit 5f8356ebd39041673b5546d77bc202295550fbe1
Author: jkctaiwan <guanru0919@yahoo.com>
Date:   Fri Mar 29 14:39:16 2019 -0700

    explicit on verb and header

diff --git a/ivi/tektronix/tektronixMDO4104C.py b/ivi/tektronix/tektronixMDO4104C.py
index 3ba6fbd..dcec4e3 100644
--- a/ivi/tektronix/tektronixMDO4104C.py
+++ b/ivi/tektronix/tektronixMDO4104C.py
@@ -70,10 +70,10 @@ class tektronixMDO4104C(tektronixMDO4000, tektronixMDOAFG):
         self._write(":data:width 2")
         self._write(":data:start 1")
         self._write(":data:stop 1e10")
-        # eanble verbosity
+        self._write(":VERBose ON")^M
         self._write(":HEADer ON")
...rest of commit...

PS C:Temp\python-ivi>


[alm]

Sure, you could also do that with separate copies (clones) of the repositories. But what's the point? Why not add the other repo as a remote and merge/rebase on that?