Attending
Ivan, Heather, Roland, Ed, Hannah, Alex, Scott, Christos
Notes:
0. Agenda bash
1. Hackathon planning - https://wiki.refeds.org/pages/viewpage.action?pageId=44959235 <https://wiki.refeds.org/pages/viewpage.action?pageId=44959235>
Currently about 24 people signed up; no sense of distribution.
Maybe have some “how to” stations to help people learn how to connect, how the various software stacks work.
We do know there will be some people there specifically for thiss.io, some for OIDC federation, some SimpleSAMLphp
For Satosa, should have a set up with a ready-made Satosa deployed, then look into how to configure microservices, and develop new microservices. Also do something internal data for Satosa. On the list, one suggestion was to see that the MFA profile passed from the front end to the back end. That might be a good project.
Ivan is still planning to work on some machine images this week and other items in prep for the Hackathon.
If anyone else has specific ideas of things to share or things they plan to prepare, please let Ivan know.
There may be some work on pySAML2 as part of the integration with Satosa (see in particularhttps://github.com/IdentityPython/pysaml2/issues/586 <https://github.com/IdentityPython/pysaml2/issues/586>)
For the OIDC Federation, there will be a couple of folks doing some java programming, and hopefully the SimpleSAMLphp will join. Roland will have a server set up to work against. Also has a few people interested in writing mobile clients, though they are not going to be at the hackathon. Roland will also set up a server for them to test against on the web. Giuseppe de Marco is working on an OIDC IdP implementation based on Django, though he will not be at the hackathon.
2. GitHub review
a. pySAML2 - https://github.com/IdentityPython/pysaml2 <https://github.com/IdentityPython/pysaml2>
b. Satosa - https://github.com/IdentityPython/SATOSA <https://github.com/IdentityPython/SATOSA>
https://github.com/IdentityPython/SATOSA/pull/252 <https://github.com/IdentityPython/SATOSA/pull/252> - (Scott’s PR) Doing things with LDAP. Are just about at the point that this can be merged. Ivan suggests that we need to normalize the attributes received from LDAP, which means removing the options part from the attributes names and then map them to other attributes. Scott notes that the attribute options need to remain; sometimes the attributes options are enumerated, sometimes they aren’t, and how they are mapped will vary (but they must be mapped). That makes this all fine, then, and the PR can be merged.
https://github.com/IdentityPython/SATOSA/pull/275 <https://github.com/IdentityPython/SATOSA/pull/275> - (Hannah’s PR) converting logging to something different, to allow outputting of structured information. Ivan will be merging changes as they come in. When we’re done, we will look at exactly what we log, whether they are the right things to capture and if they help us enough in knowing what’s happening within the service.
While Ivan was on holiday, discussed how to maintain state between back ends and front ends (use cases, not solutions). Goal is to have access to the configuration of something, like a microservice. Those values could then be used. This might be best dealt with by using handles. We have multiple back ends and front ends, and if we say we want to have access to those, there needs to be specific coupling. We will need to mimic interfaces within the backend (which they don’t otherwise have). Better to communicate data than to have functions be passed around, because the data can be served and be agnostic to the things consuming them. They become the API. Given this data needs to be communicated between requests, right now that means info has to be pushed into the state cookie. Worried about the size of that growing over time, Need alternative ways to keep state. If we use something like the web storage or the index db, then much of what we need to transfer can be eliminated. If we could have context around configuration that could be serialized, we could keep the state of that. Every time it is needed, it could be rebuilt using the serialized state. (This would be a future architecture possibility)
https://github.com/IdentityPython/SATOSA/pull/275 <https://github.com/IdentityPython/SATOSA/pull/275>
https://github.com/IdentityPython/SATOSA/pull/273 <https://github.com/IdentityPython/SATOSA/pull/273> - before merging this, note there are many common things between the way the OIDC services do things, and perhaps we should have a base for the services.
c. pyFF - https://github.com/IdentityPython/pyFF <https://github.com/IdentityPython/pyFF>
i. pyFF metadata downloads
d. …
3. AOB
Neils van Dijk reported a security issue.
The issue is with the handling of the redirect_uri. When a request is received, SatoSa validates if the redirect_uri is valid semantically and is allowed as defined in a client database.
If that is not the case, SaToSa shows an error page for example "Something went wrong: Redirect uri 'foobar' is not registered"
However, in case the parameter contains e.g. javascript like redirect_uri%3Cscript%3Ealert(%22hi%20SaToSa%22)%3B%3C%2Fscript%3E SaToSa does detect correctly this is not a correct URI, but then continues to present the javascript on the error page unmodified, hence executing the javascript.
Any other error in the url paramaters passed between rp and op should normally result in a response back to the RP without any user screens involved, however as per https://openid.net/specs/openid-connect-core-1_0.html#AuthError <https://openid.net/specs/openid-connect-core-1_0.html#AuthError>, an issue with the redirect_uri is the sole exception :(
I think SaToSa should always escape the html characters in the error message and perhaps also detect e.g. the presence of a bracket as an illegal character.
Another security issue: Scott and Ivan have discussed that the default for the SAML backend will consume the attributes without checking the scope. This might best be part of the core in pySAML2, and keeps with the most conservative defaults as possible.
Hello folks,
The UK federation team have discovered that a pyFF deployment is making a large number of metadata aggregate downloads from our Metadata Publication Service. In August, 34.74.200.81 made over 3,000 gzipped downloads of our metadata, downloading 36GB of metadata. As we update metdadata once per day, this deployment is clearly downloading excessively.
The IP address 34.74.200.81 is somewhere in Google cloud (81.200.74.34.bc.googleusercontent.com.) and the user agent string is pyFF/1.0.1. Is this anyone's deployment?
We also recommend that metadata downloaders make use of the conditional GET. Is it possible to configure pyFF to do conditional GET for metadata downloads?
Thanks,
Alex
—
Alex Stuart, Principal technical support specialist (UK federation)
alex.stuart at jisc.ac.uk
UK federation helpdesk: service at ukfederation.org.uk
Jisc is a registered charity (number 1149740) and a company limited by guarantee which is registered in England under Company No. 5747339, VAT No. GB 197 0632 86. Jisc’s registered office is: One Castlepark, Tower Hill, Bristol, BS2 0JA. T 0203 697 5800.
Jisc Services Limited is a wholly owned Jisc subsidiary and a company limited by guarantee which is registered in England under company number 2881024, VAT number GB 197 0632 86. The registered office is: One Castle Park, Tower Hill, Bristol BS2 0JA. T 0203 697 5800.
Hello everyone,
Not many people gathered today for the usual biweekly call. We decided
to postpone the call for next week. I'll followup with an agenda
notification, soon.
Cheers,
--
Ivan c00kiemon5ter Kanakarakis >:3
Hi,
I want to run pyFF using the RedisWhooshStore to leverage the superior
indexing that Whoosh provides to support API search.
The example API runner script at
https://github.com/IdentityPython/pyFF/blob/master/scripts/run-pyff-api.sh
uses this command:
gunicorn \
--preload \
--log-config logger.ini \
--bind 0.0.0.0:8000 \
-t 600 \
-e PYFF_PIPELINE=$pipeline \
-e PYFF_STORE_CLASS=pyff.store:RedisWhooshStore \
-e PYFF_UPDATE_FREQUENCY=300 \
--threads 4 \
--worker-tmp-dir=/dev/shm \
--worker-class=gthread pyff.wsgi:app
My testing shows that combination of arguments will not work. If you
start from a new deployment the Whoosh index is never generated.
The reason is the --preload option to gunicorn. When that option is used
the application code (pyff.wsgi:app) is loaded/evaluated before the
worker process is forked. As such the APScheduler BackgroundScheduler
instance is created before gunicorn forks to create the worker process.
It is realized as a thread running in the parent.
The forked child process does not inherit threads (other than the main)
from the parent. So the BackgroundScheduler only runs in the parent,
which seems initially to be the desired behavior--you would not want two
copies of the BackgroundScheduler, one in the parent and one in the
child.
Further, with --preload it is the parent process that adds or schedules
the "call" job to run the update/load cycle. Since the
BackgroundScheduler also runs in the parent it immediately "sees" the
added "call" job and runs it. The "call" job does an HTTP GET to
localhost to cause the update/load cycle.
By this point the parent has forked to create the child, and it is the
child worker that services that GET call. As part of servicing that GET
call the child worker eventually schedules the job
"RedisWhooshStore._reindex".
But since it is the child and not the parent that schedules the reindex
job, the BackgroundScheduler thread running in the parent never "sees"
that the reindex job has been scheduled. If using the default memory
scheduler job store the reindex job will never be seen, and hence the
Whoosh index will never be created.
If the command above is changed to include
-e PYFF_SCHEDULER_JOB_STORE=redis
then the job is stored in Redis. While that helps because now the
BackgroundScheduler thread in the parent will "see" the reindex job, it
will not "see" it until it wakes up to run the "call" job. So the
reindexing does not happen until PYFF_UPDATE_FREQUENCY later. I want to
run with update frequencey of one hour, but I do not want to wait an
hour for the initial Whoosh index to be created, nor do I want any
changes from the update/load to not appear in the index until an hour
later.
I think the right workaround for now is to NOT use --preload. In this
scenario the BackgroundScheduler thread runs in the forked child process
and so it "sees" both the "call" and "reindex" jobs as they are
scheduled. The Whoosh index is created immediately after the update/load
cycle.
The downside of this approach is that gunicorn should only be run with a
single worker (the default). When run with more than one worker there
would be multiple BackgroundSchedulers doing the same work, and that is
probably not desireable.
One gunicorn worker with multiple thread should be able to easily serve
the loads I expect for most of my deployments, but if pyff:wsgi:app
is to really scale I think the approach with APScheduler needs to be
redesigned. I think the scheduler will need to run in a dedicated
process that is managed via some type of remote procedure call, or
another background task/job manager will need to be used instead of
APScheduler.
Leif, do you agree with this assessment?
If so, I will submit a PR for run-pyff-api.sh that removes the --preload
option and explicitly uses "--workers 1", and that includes a comment
about only using one worker.
Thanks,
Scott K
Hey,
Sorry for the crosspost...
After a few weeks of spending all of my available development bits on
the various parts of RA21 (cf github.com/TheIdentitySelector, yes its
all nodejs!) I'm back to working on pyFF for a bit.
Here is what I have planned for in the quite near term:
1. merge the api-refactory branch which includes a pyramids-based API
2. merge documentation PR from Hannah Sebuliba (thx!)
3. tag and release the last monolothic version of pyFF
4. in HEAD which becomes the new 1.0.0 release:
- remove all frontend bits (old discovery, management web app)
- pyffd will now start pyramids-based API server
- wsgi will be available/recommended
- create a new "frontend app" as a separate webpack+nodejs project
- create docker-compose.yaml that starts pyffd (API) + frontend app
5. tag and release 1.0.0 thereby moving pyFF over to semantic versioning
After 4 it makes sense to talk about things like...
- new redis/#nosql backends
- work on reducing memory footprint
- pubsub for notifications between MDQ servers
- more instrumentation & monitoring
- adaptive aggregation for large-scale deployments
- elastic search
- management APIs for integrated editing of local metadata
- OIDC
- generating offline MDQ directory structures (cf scripts/mirror-mdq.sh)
Thoughts etc are as usual welcome.
Cheers Leif
I drafted a new consent service based on Django:
https://github.com/identinetics/simpleconsent/blob/master/README.adoc [1]
I weighted the complexity of CMservice, its lack of documentation and community support vs being an already deployed project. I think that I will drop CMservice and go ahead with developing simpleconsent in second half of September, unless someone would propose an alternative.
Any encouragement or dissuation? A consideration is that Django does not work with SQLAlchemy, which is a different type of ORM. But I would need to stick to Django for development speed.
- Rainer
[1] @Heather: Is there an RFC that dismisses the use of „simple“ in project names? My excuse is, that SCAR (Simple Consent for Attribute Release) did not sound well, either.
> Am 2019-08-15 um 20:16 schrieb Rainer Hoerbe <rainer at hoerbe.at>:
>
> Thanks for the quick answer. I hope that we can cover this in the idpy call next week, as I will be on vacation fro 2weeks afterwards.
>
> I would be interested in your assessment of the code. On my side, I am unhappy that the APi is undocumented and has to be reverse engineered from the view definitions etc.
>
> - Rainer
>
>
>
>> Am 2019-08-15 um 20:06 schrieb Christos Kanellopoulos <christos.kanellopoulos at geant.org <mailto:christos.kanellopoulos at geant.org>>:
>>
>> Hi Rainer
>>
>> We have done some further work on the CM service and we have fixed various bugs. Now not myself and Ivan are on holidays. Next week we will not be back and share the updated code.
>>
>> Having said this, we are seriously thinking to abandon this code base and develop a cm l component from scratch.
>>
>> Christos
>> From: Rainer Hoerbe <rainer at hoerbe.at <mailto:rainer at hoerbe.at>>
>> Sent: Thursday, August 15, 2019 8:22:13 PM
>> To: Christos Kanellopoulos <christos.kanellopoulos at geant.org <mailto:christos.kanellopoulos at geant.org>>
>> Subject: Re: CMservice gitlab export
>>
>> Hi Christos,
>>
>> The integration of CMservice into SATOSA is again on the top of my todo list. When I added your tar-ball from 22. May, I notices that the unit tests have not been updated to reflect the changes in src. I fixed this in https://github.com/its-dirg/CMservice/pull/11 <https://github.com/its-dirg/CMservice/pull/11>, and a few dependency issuers.
>>
>> Is there any new status on the GÉANt branch of the project? Any new commits? I would like to know if there is a chance to consolidate efforts wrt this project. do you know, or do you know someone who might know?
>>
>> Cheers, Rainer
>>
>>> Am 2019-05-22 um 12:02 schrieb Christos Kanellopoulos <christos.kanellopoulos at geant.org <mailto:christos.kanellopoulos at geant.org>>:
>>>
>>> Hello Rainer,
>>>
>>> find it attached. Yesterday afternoon, became really late night.
>>>
>>> Christos
>>>
>>> On 22 May 2019, at 11:54, Rainer Hoerbe wrote:
>>>
>>> May I send a friendly reminder?
>>>
>>> thanks!
>>>
>>>> Am 2019-05-21 um 08:47 schrieb Christos Kanellopoulos <christos.kanellopoulos at geant.org <mailto:christos.kanellopoulos at geant.org>>:
>>>>
>>>> Hello Rainer
>>>>
>>>> I am at the hospital, but I will be able to send it to you later this afternoon
>>>>
>>>> Christos
>>>>
>>>> From: Rainer Hoerbe <rainer at hoerbe.at <mailto:rainer at hoerbe.at>>
>>>> Sent: Tuesday, May 21, 2019 9:46 AM
>>>> To: Christos Kanellopoulos
>>>> Subject: CMservice gitlab export
>>>>
>>>>
>>>> Hi Christos,
>>>>
>>>> You mentioned in the last idpy meeting that I might get a copy of Geant’s CMService repo on gitlab. Whom would I ask to get it?
>>>>
>>>> Thanks and best regards
>>>> Rainer
>>>
>>>
>>> --
>>> Christos Kanellopoulos
>>> Senior Trust & Identity Manager
>>> GÉANT
>>> M: +31 611 477 919
>>>
>>> Networks • Services • People
>>> Learn more at www.geant.org <http://www.geant.org/>
>>>
>>> GÉANT Vereniging (Association) is registered with the Chamber of Commerce in Amsterdam with registration number 40535155 and operates in the UK as a branch of GÉANT Vereniging. Registered office: Hoekenrode 3, 1102BR Amsterdam, The Netherlands. UK branch address: City House, 126-130 Hills Road, Cambridge CB2 1PQ, UK.
>>>
>>> <cm-service-devel.tar.gz>
>
Attending:
Scott Koranda, Heather Flanagan, Leif Johansson, Giuseppe de Marco, Johan Lundberg, John Paraskevopoulos, Alex Stuard, Hannah Sebuliba,
Regrets:
Ivan, Rainer
Virtual IdP front end to Satosa - can expose multiple virtual IdPs through the Satosa front end. Can configure various options for the IdP, including the name of the IdP, the scope the IdP wants to use, etc. That config belongs to the front end. Scott also wants to have some microservices that operate on the assertions as they go through the system, and the microservices should have the same access to that configuration (e.g., so they can see the scope). Waiting on a decision from Ivan on how to implement this.
Can you run a single Satosa instance with multiple front and back ends? Example, a SAML back end that would authN against eduGAIN, and another that authN to ORCID, and front ends that would work with either SAML or OIDC.
Question: has anyone set up OIDC front end and had it work with mod_auth_oidc? Mod_auth_oidc is complaining. Giuseppe is planning on doing this in the next month or so.
With the OIDC front end, it won’t automatically work with multiple backends (cannot select between multiple backends). Need a custom routing service. Does anyone have such a routing service available? Giuseppe wrote one; can find it in the Satosa PRs. It intercepts the call and uses a map of entity IDs that need this.
Update on pyFF
Current release = 1.1.1; there are some bug fixes that need to go into 1.1.2 asap.
Code is stabilizing, but not sure he’d bet on 1.1.2 being stable.
2.0 will start with Leif removing the front end bits; he will provide a bash script to help people who are used to calling pyffd. There will still be a wsgi app (and it will be the main entry point).
Hannah has been working on some interesting memory things. She is looking for memory leaks. Scott thinks that when pyff is running as a server, it needs to never create a really large DOM, avoid ever having read the eduGAIN feed as a single DOM object, because that creates a huge, unnecessary memory request. Also have to avoid creating lists of many things; even if you don’t load the whole DOM, you have a list of small DOMs and if it is held in memory before being given to the backend store, you still consume a lot of memory. Scott suggests the architecture needs to shift from parsing large chunks of metadata, to parsing small chunks, handing them off to the backend, then garbage collect. Leif points out that as soon as you’re dealing with signed metadata, you have to handle all of it at once. Could try to do something by making the pipeline smaller.
One suggestion: switch to the Redis backend. Which could work in some use cases, but not for the full aggregate
Could do an offline fetch as another way to control size.
One goal is to keep pyFF from needing a server with more than 4GB. Not likely that’s going to function as eduGAIN gets larger
In eduTEAMS, pyFF does take up the largest memory footprint.
Could start pyFF, ingest all you need, use mirror MDQ to produce an offline copy, then shut down the pyFF service until you need to reingest. The offline MDQ could be used for discovery for as long as the signature is valid. Can use the thiss.io MDQ (thiss-mdq <https://github.com/TheIdentitySelector/thiss-mdq>) for a miniature search function.
Giuseppe uses pyFF with a scheduler.
Another alternative is to use the default discovery service being put together by SeamlessAccess.org <http://seamlessaccess.org/> (based on RA21)
Would be interesting to compare woosh+Redis to a JSON-only index store. Action item for Hannah.
How to exclude entityIDs from pyFF? It works as expected up to 0.9.3. Can use a filter, but the previous version of fork, merge, remove does not. (The latter impacts the current working document, and should not actually work.) Suggest you look at load-cleanup - there’s a way to run a pipeline early on before you update the backend store, and that might be it.
