Hi all,
This is what I remember from last Monday's meeting. It was Mikael, Ivan
and me, please both of you correct me on anything I misremember or
forget.
There was talk about pyFF's issue 291 [1] (Broken handling of # in
filenames and urls), Mikael said he was working on it - (or already had
a PR fixing it?)
Then Ivan told us he had released a couple of pysaml2 versions, and was
working on a further couple of PRs, but I forget the details, perhaps
you @Ivan can fill them in.
Finally there was some discussion on pyFF's issue 289 [2] (When an
entity is loaded from 2 sources, entity data from the 1st source is
lost). I started by showing some "proof of concept" code (just around 15
lines, see [3]) that addresses this issue. This is just a POC since:
* The issue is only addressed in MemoryStore, it would probably also
need to be addressed in RedisWooshStore.
* The data is duplicated. In the current version, when entity data is
loaded from the sources, it is kept in 2 structures in the store:
`md`, which is a dictionary of md sources to lists of entityID's, and
`entities`, which is a dictionary of entityID's to entity data. In the
POC code, we add `md_entities`, which is a dictionary of md sources to
dictionaries of entityID's to entity data.
In the POC, the first 2 structures are still used for all the purposes
that they have ever been used for, and the new `md_entities` structure
is only used when the select pipe is configured with the new option
`dedup False` (the option defaults to `dedup True`).
But of course `md_entities` contains all the data that is in `md` and
`entities`, so we might think of removing the latter and using the
former for the purposes that the latter are used.
A few concerns with this issue were discussed:
* How do consumers (MDQ service) deal when they have duplicates in the
metadata and are asked for some particular entity? In thiss-mdq, when
the metadata is loaded, entities are deduplicated, but some number of
(multivalued) entity attributes are merged.
* What happens when one entityID present in 2 md sources correspond to
different entities (name collisions)? This is a difficult problem, but
somewhat orthogonal to the issue, since in the current pyFF form, it
is also present (currently, one of the entities would just dissapear).
* Can this be abused, if federation A has less strict requirements for
some entity attribute than federation B? Yes possibly, this would need
some risk assessment by the working group; some of the metadata would
not be affected, for example registrationAuthority.
In the end, we agreed that more discussion is needed to reach a
difinitive conclussion, and that any solution is going to carry problems
that can at most only be mitigated but not fully solved.
Best regards,
1.- https://github.com/IdentityPython/pyFF/issues/291
2.- https://github.com/IdentityPython/pyFF/issues/289
3.- https://github.com/enriquepablo/pyFF/commit/0fb326d6043c1a3c6c2bb9a431cf4a9…
--
Enrique Pérez Arnaud
Hi
I have produced a minimal POC that would allow pyFF to load metadata
from different sources and produce a list of entities in which there may
be more than one entity with the same entityID (one for each source in
which it was present). It can be tested with the attachments in the
issue tracker.
1.- https://github.com/enriquepablo/pyFF/commit/0fb326d6043c1a3c6c2bb9a431cf4a9…
Best regards,
--
Enrique Pérez Arnaud
Attendees: Johan L, Shayna, Ivan, Mikael, Matthew, Enrique, Hannah
0 - Agenda bash
1 - Project review
a. General
- Moving project repos - all should be moving under IdPy, but who will
maintain the ones that are new and not going to be under other projects?
- Mikael will keep them floating - most (other than the ones being
added to Satosa) are considered POC, and Sunet and SWAMID will
use them as
reference.
- Mikael will look into the process for bringing the repos under the
IdPy umbrella, described here:
https://github.com/IdentityPython/Governance/blob/master/idpy-projects.md
b. OIDC libraries - https://github.com/IdentityPython (idpy-oidc,
JWTConnect-Python-CryptoJWT, etc)
- Nikos will be putting up PRs with some new functionality.
- Everything should be under Roland's branch for the new repos
c. Satosa - https://github.com/IdentityPython/SATOSA
- Will be posting a new release after the call with the
ldap_attribute_store plugin updates.
d. pySAML2 - https://github.com/IdentityPython/pysaml2
- Will be creating a new release after the call to include:
- https://github.com/IdentityPython/pysaml2/pull/964
- https://github.com/IdentityPython/pysaml2/pull/897
- uses pydantic v1 but now we have pydantic v2, so want to make
sure there are no problems - there may be an issue with
the python version.
Ivan is testing with 3.13. Mikael knows there is a
breakage with pyFF with
3.13 that he thought might be related to pydantic.
- Next will look at some changes Giuseppe has prepared and is
using in his fork around namespace names.
- https://github.com/IdentityPython/pysaml2/pull/625
e. Any other project (pyFF, djangosaml2, pyMDOC-CBOR, etc)
- pyFF: Mikael will be taking a look at the hashmark issue mentioned in
the last meeting. Ivan is looking into this as well.
- Mikael and Enrique are collaborating on the issue Enrique described
last week.
2 - AOB
- Matthew had posted some things on Slack about the attribute mapper,
but was able to figure out what he needed.
- SAML defines attributes - they are not just an identifier. There is
the name, the friendly name, and the name format. The name
format tells you
how the name is structured - it is not really a string. It could be a url
or uri , for example. Within the name you could have a uri with a hash
symbol with a pointer, so you cannot just compare the values as strings.
Parsing the objects the right way may show they are the same. The
uniqueness of an attribute does not come from the name - you have to
combine it with the name format.
- Ivan will try to answer this on Slack and give some examples
- Matthew is currently working on signing outgoing SAML requests - it
is not working out of the box. He will gather his questions on this for
another time.
- Matthew is also working on how to structure tests for an application
that uses SAML, and uses jwts after the SAML response. Would like to mock
up a real world application.
- Next goal is to be able to do integration testing, deploying an IdP
that facilitates that.
- Also doing all the same stuff with open id connect. Still working
on getting the proper configuration.
- Next week, Shayna will be out and Matthew has volunteered to take
notes.
Attendees: Johan L, Shayna, Ivan, Mikael, Matthew, Enrique, Hannah, Alex
0 - Agenda bash
1 - Project review
a. General
- Discuss whether we should move some of Roland's project repos (which
have been moved under Sunet) to be under IdPy:
- SUNET/openid4v - ??
- SUNET/satosa-idpy - frontend - becomes part of SATOSA
- SUNET/satosa-openid4vci - extension to proxy - becomes MR under
SATOSA
- SUNET/fedservice - under idpy
- SUNET/idpy-sdjwt - under idpy
- Need time to understand - should probably slack Roland about
what changes they are introducing
- Also there is a document that explains the process for adopting the
code under IdPy which Ivan was going to share.
b. OIDC libraries - https://github.com/IdentityPython (idpy-oidc,
JWTConnect-Python-CryptoJWT, etc)
- New MR(s) with Nikos introducing changes around resource indicators/
audience policies. The internal fork is also being synced with the
upstream.
c. Satosa - https://github.com/IdentityPython/SATOSA
- trying to get to release for LDAP plugin changes
d. pySAML2 - https://github.com/IdentityPython/pysaml2
- Want to make sure latest releases will work together (both Satosa and
pySAML2)
- trying to get to release for typing changes for the entity
categories
e. Any other project (pyFF, djangosaml2, pyMDOC-CBOR, etc)
- Enrique mentioned in issue in pyFF, where entities in different
federations have the same entity id. One possibility is that it
may be the
same entity registered to different federations, but also it could be two
different entities in different federations but they have the same entity
id. Enrique has the use case in Seamless Access, where entities are
registered in two different federations, in different entity
categories. He
needs to be able to merge certain entity attributes in this case.
- The issue and Enrique's notes on it can be found here:
https://github.com/IdentityPython/pyFF/issues/289
- The way things are now, the entity in the first federation would
be overwritten by the one in the second federation.
- We may want to merge different entity categories, which may
differ from one federation to another.
- For filtering in mdq, we would need the attributes, which could
have different values in different federations, to be merged.
- Enrique's PR was merged to allow select pipe to be able to
produce a list of entities with duplicates. But the
overwriting happens
before the select pipe - it happens in the load pipe - all
the entities
from all the sources are squashed into one dictionary, keyed
by entity id.
- One idea would be to keep the entities keyed not only by entity
id but also by metadata service in the load pipe, but there
is some concern
that this could introduce problems. When the user requests an
entity id,
how would we know which one to choose? Could this be handled
in the MDQ
service? Would the rest of the attributes give a fairly good
heuristic to
distinguish between those cases? The select pipe can default
to doing it
the way things currently work - overwriting the first with
the second - or
use a heuristic downstream to either merge the duplicated
entities or keep
them separate since they are different. Mikael warns that we
need to be
careful not to change the default heuristic - otherwise we
might break all
federations. A new heuristic would need to be approved for
each aggregator.
We need to better define what happens when we get conflicts
so someone will
be notified, instead of just getting the last loaded. We will need a
feature flag so we don't break deployments.
- Ivan wants to establish if this a current problem, or if we are
trying to think ahead to a potential problem, especially in
the case where
the entity ids are the same in two different federations but
the actual
entities are different. We should document that we make the
assumption that
there will not be different entities in different federations
that will
have the same entity id. However, If they are the same entity but in
different federations, there is an issue with the entity
exposing different
things in each federation.
- Merging multi-value attributes: What if there is an entity with
an entity category in InCommon (say, "hide from discovery")
that is not
available when that same entity is in eduGain? If we merge
them, something
that was not intentional would happen. Need to examine these
things more
thoughtfully before doing merging - some are based on policy.
- PyFF - Microsoft EntraID causes a problem with hash marks in
entity ids. pyFF doesn't support getting it by url, only by sha1sum. pyFF
refuses to load a file when hash marks appear in it.
- pyMDOC-CBOR
- will create an MR around a request for an improvement
2 - AOB
- Welcome to Alex - he has been working with Ivan looking at the
proxy/Satosam, focusing on projects under GÉANT CoreAII (formerly
eduteams). He has a lot of experience with different languages and
libraries, and has a lot of ideas on how we can approach things and
structure things better.
- Need to create a list of questions generated from Matthew's efforts to
write his own IdP / test SP, then work through a few of them at a time
weekly.
Enrique Pérez Arnaud has asked me to draw attention to notes he added to a
pyFF gihub issue, regarding duplicate entity ids in pyFF in different
federations. The notes can be found here:
https://github.com/IdentityPython/pyFF/issues/289#
I will also add this link to the discussion of this issue in the IdPy
developers meeting yesterday. The notes from that meeting will be out later
today.
Thank you,
Shayna Atkinson
SCG
*Idpy meeting 27 January 2025*
Attendees: Johan L, Shayna, Ivan, Mikael, Matthew, Enrique, Hannah
0 - Agenda bash
1 - Project review
a. General -
- Documentation pain points run-through / mock saml workflow / mock oidc
workflow
- Matthew has a github repo that mocks up a SAML authentication flow
using pytest. Pysaml is needed for a client to write more
than one identity
provider, and a service provider to test those identity providers -
https://github.com/xenophonf/mock-saml-flow
- Has a configuration for IdP and SP from his Satosa
configuration.
- There is some documentation for the configuration, but one
challenge is the configuration is entirely a mapping. There's
no typing
hints, things aren't discoverable in his development environment.
- Doesn't want to get into Flask, just wants to concentrate on
using what paysaml2 calls a Server and a Client. However, there is no
documentation on the classes and methods to use.
- No API documentation
- the simple examples included in code base just tell you how to
run it; not how to use the code
- one option is to go to the example code and try to figure out
how to develop an SP or IdP, but reading the source code is
really hard.
- what is repoze?
- it's not clear where to start. First guess is to start in idp_
uwsgi.py, skimming for things like url routes (where do
authentication requests come in/go out?). Only able to
find a SSO class.
How is this invoked?
- Conclusion: it takes too long to figure out how to find the
basics of what you need to know.
- Another option - test suite. Not successful here either.
- Next option - reading through Satosa code. Found Server class.
There are no type annotations, no document string for the
method to create.
There is some document strings for the other functions, some
params, some
information, but not enough to know what to use when. Found the
create_authn_response() method but there is no information on
what should
be provided for a good saml response or what structure it
should follow -
the only way to get this information is trial and error.
- idpy-oidc documentation looked more promising
- Encouraged by newer style for the documentation page.
- Wanted to write a small RP - looked at client documentation
- There is a workflow, describing the process and a high level
overview of how oidc RPs work. Very helpful.
- API documentation is lacking. No method signatures. For
example, is issuer_id the only argument the begin() method
takes? But the
Tier1 API design is good.
- No examples on how to instantiate the IdP
- Left reading through the source code again, trying to intuit
from example code and test code how to do it.
- When looking to write code to have a test OP (to test RP) -
Server code - the documentation only tells you how the
configuration
directives work. No API calls, no example code, no examples of
instantiation.
- SQLAlchemy ORM is an example of great documentation on how to
get going - type annotations, explaining what each parameter
is for, etc.
- For mock saml - took 2 months to write 179 lines of code
- Kushal has tried to document some examples -
https://kushaldas.in/learningsaml - unfortunately this starts from
a place beyond where Matthew's knowledge is
- Ivan is the person Matthew and Hannah will need to talk to to
get their project going. First focus on SAML backend for Service, SAML
Frontend for Idp. This won't give the complete picture but
it's a start.
- For Service:
https://github.com/IdentityPython/SATOSA/blob/master/src/satosa/backends/sa…,
especially authn_request and authn_response
- For an IDP:
https://github.com/IdentityPython/SATOSA/blob/master/src/satosa/frontends/s…,
esp handle_authn_request and _handle_authn_response
- Ivan acknowledges there is not developer documentation
- initial step to address docstrings
- Need to define public API methods and documentation - this
would 90-95% of what you can do with SAML
- The examples are old, and sometimes create confusion (for
example, the Server in pysaml2 is an IdP, the Server in
examples is a wsgi
server, the Server in Satosa is a proxy server).
- A referenced example against a SAML trace would be very
helpful
b. OIDC libraries - https://github.com/IdentityPython (idpy-oidc,
JWTConnect-Python-CryptoJWT, etc)
c. Satosa - https://github.com/IdentityPython/SATOSA
- LDAP plugin release coming
- Matthew will have a new SATOSA Docker image out new week
d. pySAML2 - https://github.com/IdentityPython/pysaml2
- release coming to address xml enc changes and introducing types around
entity categories from Frederik and Johan.
e. Any other project (pyFF, djangosaml2, pyMDOC-CBOR, etc)
- Roland's code has been moved under Sunet - mostly related to openid
federation and credential/wallet. Let's have a discussion whether some of
these need to be moved under IdPy
- SUNET/openid4v
- SUNET/satosa-idpy
- SUNET/satosa-openid4vci
- SUNET/fedservice
- SUNET/idpy-sdjwt
2 - AOB
3 - Action items
- pull out questions Matthew raised one by one and document them
- For example some classes were machine generated based on schema
changes (Matthew suspected this) - perhaps we should add docstrings to
explain how/when this was don
