Steven R. Brandt: And we all look forward to hearing it take it away Eric Thank you.
Erik Schnetter: it's a pleasure to be here it's a pleasure to talk about carpet X i'm going to give a presentation explaining how to use crashlytics and some of the basic concepts there.
Erik Schnetter: And this is meant for early adopters it's almost ready for production, use there's one feature that isn't there yet, and that is checkpoint in recovery from that you should be able to use it.
Erik Schnetter: So this presentation has two parts in the beginning a bit of an overview of what is in carpet X and the basic concepts.
Erik Schnetter: And the second part of a bit more hands on investments i'm showing some actual code and walk you through writing your own thorn that uses carpet X pointing out the differences to carpet.
Erik Schnetter: So, and the basics carpet acts as a new driver for the Einstein tool kit it's based on amex.
Erik Schnetter: And when we started out with the project we had hopes to get the following features from corporate amex first amex has many users independently funded, so we benefit from their development don't have to do everything ourselves.
Erik Schnetter: It provides self centered centered vertically centered grits etc properly flexing.
Erik Schnetter: This is very good for hydro dynamics currently I hear the people who do big IQ simulations don't like live mesh refinement boundaries.
Erik Schnetter: inside a star because it leaves the large areas The hope is that reflecting makes this possible, because it has much less error than profiling boundaries when there's, for example, a shockwave going through that.
Erik Schnetter: it's once a month, based on docu criteria, so you don't need to say where you put your boxes rapid you say which error each cell has and then depending on a special.
Erik Schnetter: On rx who finds or doesn't define it has support for accelerators, for example, kuda but also we have for other things, and that of course is interesting for us.
Erik Schnetter: The elliptic solvers built in that has long been missing from the Einstein toolkit louisa Thank you and yet, but on that, but for some reason have rock hasn't been picked up.
Erik Schnetter: Maybe that gives that and other boost and i'm access to scalable it supports excess scale that's what the X and X is about so let's see what's that about.
Erik Schnetter: And we had two other girls in the project.
Erik Schnetter: and the first is to simplify this calculator if you have a measure of i'm in code and schedule, then you say rich routines happening time global mode level mode local mode, etc.
Erik Schnetter: it's quite easy to get confused and to make an arrows arrows are not open detected, you have some undefined points difficult to track down, we wanted to be much more rigorous about that and I O performance also was a bit of a problem in particular.
Erik Schnetter: i'm sorry.
Steven R. Brandt: i'm sorry sorry just a minute Peter, are you recording this did you hit record.
Erik Schnetter: Yes, Peter is recording I heard the message earlier.
Erik Schnetter: At the moment, was more.
Eloisa Bentivegna (IBM): Attack there's a tag on the top left.
Steven R. Brandt: Okay, all right Oh yes, I see it now alright sorry.
Erik Schnetter: Very good, thank you.
Erik Schnetter: So this is the current state of affairs, the green things are working independently funded well, they have funding they're collaborating with us they're happy to do so.
Erik Schnetter: they're self face but accented grits etc flexing we tested it with a simple non relativistic icon, so this is all done and simplify the scheduling preventing arrows and so on, but it's all working beautifully.
Erik Schnetter: I think so still work in progress, for example, a Mr based on Doc criteria, but all the game itself works, the question is, what kind of refinement criteria you actually want to apply.
Erik Schnetter: And then, our Community doesn't have much experience with that i'll keep it in blue in progress, instead of marking it as green as a self thing.
Erik Schnetter: Support for accelerators So this was the last feature that we worked on in the past weeks, so we can run solve the Einstein equations on a gpu that works fine.
Erik Schnetter: And it's the same code that runs efficiently on cpu and runs on gpu so that's all good but it's not yet properly optimized for a gpu so that's still work in progress.
Erik Schnetter: And elliptic solvers the problem there is that it works Nice, I was able to solve the persona equation, but they only support second order stencils know fourth order stencils.
Erik Schnetter: And it seems to be difficult to modify their code, you probably have to be right, a whole lot of operators to do that.
Erik Schnetter: also work in progress access kill scaling I ran with carpet X and up to 1500 courses nicely no problem there.
Erik Schnetter: But 1500 is an interesting you want to have at least 10 times as many course to be interesting, and at least 100 times as many to actually be interesting and that still needs to be tested I don't expect any fundamental problem there.
Erik Schnetter: and improving I O performance, so we have, of course, a new file format that comes with amex and.
Erik Schnetter: It should be more efficient, but also that hasn't really been tested at scale.
Erik Schnetter: So what else is working today in carpet X I just mentioned, we have a whole both based on html5 and ask you, I hope you can read and write html5 file is using the silo format.
Erik Schnetter: that's their native format for visit and checkpoint recovery is not working, yet that's the big thing that's missing from my oh.
Erik Schnetter: We have reductions in elections, but no hype is loving amex itself the library doesn't support, I was loving, so we have to work around that.
Erik Schnetter: That symmetries priority boundaries and reflecting symmetries but no be toting symmetries, yet they require high pass loving and there's no hyper slapping it out REX.
Erik Schnetter: And we're working on that adaptive mesh refinement brooks with higher order prolongation and restriction cell vertically centered centered centered anything you want should.
Erik Schnetter: be able to buy new wave toy your basic skill away of course that's nice there's a high tutorial that the basic non relativistic all the questions testing all the hydro stuff.
Erik Schnetter: And we can run the Einstein equations, although they haven't been Cindy personalized so I expect them to be slower than the respective mcleod and God.
Erik Schnetter: And I hope your mouth is watering and the main point of this presentation is get you interested in actually trying it out and giving feedback and making sure that all the work is actually useful.
Erik Schnetter: So here, I want to show a movie I know showing a movie is always a bit of a problem in zoom let's just see how it goes, this is just the QC zero simulation.
Erik Schnetter: You see two black holes will begin they'll do three quarters of an orbit before merging that i'm showing the labs in the beginning, the labs changes because the gauge countries and setup.
Erik Schnetter: they move around yeah there's no horizon Finder and that one and No way extraction i'm sorry.
Erik Schnetter: Right about now the black holes would merge you don't see that and there's a final Nicole and simulation continues on a bit further that should be a fully fourth order simulation.
Erik Schnetter: And I parameter file says the final solution is one over 24 the course for solution is one that doesn't make sense, because it's supposed to be a power of two but.
Erik Schnetter: Whatever so that there's a nice proof that things actually work smoothly, but there's things to do, but I would like to show side for the data wave of course not just the labs function.
Erik Schnetter: So now to the.
Roland Haas: changes.
Erik Schnetter: Between carpet and carpet X regarding adaptive mesh refinement carpet X does not yet support sub cyclic in time when we started to designing prosthetics there was.
Erik Schnetter: A lot of doubt in the carpet Community whether sub sampling, and time is worth the effort.
Erik Schnetter: A it makes the scheduling much more complicated it is really expensive what the buffer zones and everything and stability and time interpolation is only a second node accurate.
Erik Schnetter: And if you have a rather small defined region near a black hole it's really difficult to obtain scalability.
Erik Schnetter: Because the sub sampling each level needs to be adjusted diluted over the processes independently, because it serialized.
Erik Schnetter: So we started without sub cycling and time there is a much more efficient, that is really scalable it is much simpler.
Erik Schnetter: And everything happens at the same time prolongation and restriction happened when synchronizing all the levels, I was at the same time, so it's beautiful it's scalable.
Erik Schnetter: I just wonder, but whether it's actually slower in the end, because now, you need to evolve, the course kept much more often anyway be got planning to add subside in time in the future, but at the moment it's not the priority moment we want to push it towards doing some real physics question.
Erik Schnetter: Regarding is different, instead of specifying where the boxes go you define for every great cell, but the local areas and you choose a threshold.
Erik Schnetter: And cup of X splits the domain into eight by eight blocks and every block that where the arrow is too large, that is going to be refined and up to the maximum level that you specify.
Erik Schnetter: And yeah how to choose the local error that's up to you, when you write your phone.
Erik Schnetter: When you write your evolution system, there are several ideas I don't want to talk about it here, but we can just discuss the idea later, if necessary, so I know multiple systems available yet that can be added in the future.
Erik Schnetter: Right now, I want to show a bit of a code look walking through the interface that CCL i'm going to switch my screen now to get out of full screen mode here.
Erik Schnetter: and open.
Erik Schnetter: Oh i'm sorry I didn't prepare the window that I want to show.
Erik Schnetter: Please give me a minute by I opened and emacs window.
Erik Schnetter: Where I can show with the code and now I want to share something different.
Erik Schnetter: I assume you're not seeing an emacs window and Dr they're going.
Erik Schnetter: To show a thon cultivate a cpu and the interface CCL quick check that the font is large enough if not to expect you to speak up.
Erik Schnetter: Very good.
Erik Schnetter: So the interface that CCL looks slightly simpler and then with carpet.
Erik Schnetter: One thing I didn't mention before that the thorn method of lines is quite tightly coupled to the driver so at the moment there is a new version of more called O D solvers that we're using that can fix later.
Erik Schnetter: And it's more tightly integrated with a driver and uses a more modern thing, so you don't need to.
Erik Schnetter: register which functions in the beginning to see what is evolved or not product, you can add a tag to the grid function and let's say for this group, the Group is called state.
Erik Schnetter: They are right hand side variables in a group called right hand side, which is that group down here.
Erik Schnetter: And that tells the time integrator that that state group should be above the time and that this is the respective right hand side routine it simple thing then it's all written down, yet the interface, as it should be even more was designed, maybe the tax mechanism didn't exist yet.
Erik Schnetter: Also, you see that there we don't decay anytime levels because we don't need any time levels, the time integrator allocates it's temporary storage automatically knows how to do that, but there's just the current time level and everything else happens in the time integrator.
Erik Schnetter: and the last thing that you see here is this index statement carpet X because of amex support self centered face entered blah blah blah variables, so you need to specify what kind of centering you have.
Erik Schnetter: And that means in each direction well 111 just being self centered 000 would be worth like centering.
Erik Schnetter: For historic reasons decisis self centered twice if i've asked you today, it would be vertically centered the code would be very similar, and I think the fall is voting Center if you don't specify the index, but why not specified.
Erik Schnetter: And if you have a hydrophone then of course you want to specify the variable tell Center under the fluxus or face ended, so you said, one of these two zero and their respective thing and respective directions, it becomes so fake centered.
Erik Schnetter: And that's that's about it here and there is this loop thing that simplifies looping over quick functions because of other additional features that copied access that.
Erik Schnetter: i'll describe later oh and there's another tech here checkpoint equals know this essentially means the variable it's not checked pointed that that's a fancy way of saying that this is not part of the State vector.
Erik Schnetter: The State vector is a sequence essentially consists of all variables that are checkpoint it maybe we should rename it.
Erik Schnetter: Initially it was introduced to not checkpoint any variables that you didn't want to, but at the moment, we should have tags here in the interface declaring which variables are part of the State vector and whatnot you know.
William Gabella: Eric now now might be decide do we do edge Center does that even make sense for Einstein tool.
Erik Schnetter: It does it does make sense, you don't need it for hydro dynamics, but we also played with the Maxwell equations I give up because they don't know enough about the Maxwell equations to to give it stable.
Erik Schnetter: But they are you want edge centered variables and if you have magnetic fields that, then you also want to extend it and I think edge vendors have a single one and two zeros and that single one then says in which direction the he goes, you have X, Y and Z axis.
Erik Schnetter: And that is all supported by amex and we are file or application restriction of writers, all of these.
Erik Schnetter: Let me switch back to.
Erik Schnetter: The presentation it's going to be forth and back.
W. Kastaun: Sorry quick question on that it's also a sort of finance volume, for example.
W. Kastaun: With all prolongation so take into account that it's an avalanche Obama said Oh, I said just point by sapa.
Erik Schnetter: And that is what I call self centered.
Erik Schnetter: Self centered operators are to find out volume operators.
Erik Schnetter: That means, everything is an average over a cell and restriction and prolongation take this into account, so if you restrict on vertices then of course you want to have a nice curve and you interpolate in between.
Erik Schnetter: And if you restrict self centered centered values, then you need to make sure that the integral there's some over the cells also matches that gives you an additional conditions or the order of interpolation is one lower, and that is built into the respective operators.
Erik Schnetter: So, to my knowledge, the cell centering and face centering is exactly what you want, for hydro, but please tell me that i'm wrong if i'm wrong about that.
W. Kastaun: So, but I am not doing height or and want to do really sets and that finite differences, without dealing with find that volume.
Erik Schnetter: That okay that might also be pause that's also possible that different kinds of stencils what you just described is I think what I implemented first.
Erik Schnetter: And then there's a second set of sensor the different names that do the thing.
Erik Schnetter: And I tried to make everything possible I also tried to implement kind of vertex century with the leveraging.
Erik Schnetter: But I couldn't make a direct I think it's inconsistent you can't really make that work so that doesn't work so at least we cases are supported.
Erik Schnetter: there's vertex, and that is interpolation their self centered with interpolation and self centered with averaging.
Erik Schnetter: And I didn't think that somebody actually wanted to do the second case, you just mentioned the self centered with interpolation I was actually thinking of taking it out just to get around to doing to get I can keep it in.
Erik Schnetter: Now about adaptive mesh refinement more details I mentioned, it does not get support sub cycling, in time, so if you want to do binary black holes.
Erik Schnetter: With the refinements deck of 10 or 15 devils that's probably not the time, yet to do that with this infrastructure, all the levels advanced simultaneously.
Erik Schnetter: And that, of course, makes it much more efficient and much more scalable you don't need buffer zone so everything that's also a big advantage.
Erik Schnetter: And ml dimension that restriction happens venue synchronized that's a change from carpet with sub cycling you restrict that a very well defined point in time, it was the cost could.
Erik Schnetter: Define grid to the same time, and once that the same time that's when you restrict from fine to course.
Erik Schnetter: and incorporate X every time you synchronize you fix the boundaries as well, and you also fix the interior of the course quits by restricting there's a flag for that you can switch that off, but I think the current default is good but it's not quite clear whether it's actually true.
Erik Schnetter: Regarding this, I didn't show this slide again, this is a copy that I inadvertently made made.
Erik Schnetter: I might have done that it all sounds very familiar you do find the local or have a cell phone every cell towers, a grid function that carpet expert whites.
Erik Schnetter: And in that grid function you fill it with the arrow for every cell that you have, and you define a special than 10.
Erik Schnetter: carpet X sorry i'm REX would actually refine it correspond quickly.
Erik Schnetter: And there are two ways of thinking about this one is that you define an arrow for every cell and then have an error threshold and depending on the threshold.
Erik Schnetter: You define whether to refine or not, and another way to thinking about it is that you have a length scale.
Erik Schnetter: In your physics and you need to capture that link skills, you want to compare that link scale with the current grid spacing.
Erik Schnetter: And you could get a link skill, for example, by taking some physical field and taking the derivative the absolute value of the gradient but give you a link scale.
Erik Schnetter: See, if you take that define it by it, so you divide that absolute value of the gradient by typical value for the grid.
Erik Schnetter: And then you get the link scale and then you compare that to the link scale of the current level, and then you know whether you find or not and depending on that you refine or not.
Erik Schnetter: And that should be babbling more physical accurate, but I don't know what factors, you need in there, I hope that other people who have familiar with other frameworks, I of course mostly familiar carpet have ideas there and can teach us how to do that.
Erik Schnetter: I was going back in time haha that's my show this slide again I am so embarrassed please tell me these things it's not one of you watching the page numbers okay yeah, this is what I actually wanted to talk about now scheduling scheduling is slightly simpler so.
Erik Schnetter: When you set up initial conditions it actually initial conditions are set up in a loop.
Erik Schnetter: The cost good initialized then it sets up initial conditions there and then it evaluates the refinement criteria and if there's refined when needed.
Erik Schnetter: Things are fine and the initial conditions have we been until no more refined minutes needed and, in the end, as usual, you run post up analysis and output on all the levels.
Erik Schnetter: And evolution runs in a similar way well if unless you're done first you regret and then the run poster grid etc you initialize the new levels, then you evolve post up analysis output is the same for initial condition that's the general way and.
Erik Schnetter: know that the next slide is going to be a bit technical you might use a different reference.
Erik Schnetter: There are these kind of bins and here describe what the reporting scheduling bins are called or what you're supposed to be supposed to be doing in the scheduling.
Erik Schnetter: and bass grip that's a good place to set up constant data, for example, coordinates.
Erik Schnetter: yeah if you have a regular grip there's not much point in actually storing the coordinates but that you would do that and basically it is we run when a new level is created, but nothing is involved in their.
Erik Schnetter: Initial and post initial that's when you initialize the state vector and also apply boundary conditions, etc, on it.
Erik Schnetter: And I say boundary conditions, I mean synchronize balangay to restrict our boundary it's and so they're all great points are defined, when you come out of initial and post initial.
Erik Schnetter: And then they're going to pin that you will the state vector and apply boundary conditions, etc, and so forth, and, of course, if you use all the servers or or mall or something that schedule it's evolution in this able button, where it has the Roman quota cycles there.
Erik Schnetter: And post step and analysis that's where you calculate quantities that don't influence the evolution, for example, you have other constraints reasons why a scarce there, and so on.
Erik Schnetter: And this is also where you define the local arrow for recruiting for the time evolution.
Erik Schnetter: Because, after the post there been many things go back in time to the beginning of the evolution and that's when it looks at things but where things should be recreated.
Erik Schnetter: And post up is also run when you recover wants a covering is implemented after recovering data it runs postscript and just chooses, whether you want to refine before starting the evolution and there's a post rick rubin.
Erik Schnetter: A very good need to reapply the boundary conditions covered X doesn't know what fields should have bought boundary conditions, so you need to synchronize balangay restrictive like the outer boundary conditions there and so.
Erik Schnetter: Next, I want to show you the schedule of life by cpu i'm going to switch over to my Max pushing the wrong button No, sorry push the wrong button here.
Erik Schnetter: let's look at this kick your view.
Erik Schnetter: And yeah You see, at the beginning, they are no storage statements here that's just because storage management isn't implemented yet we allocate storage for all variables all the time it's quite convenient the future so many to fix that, of course.
Erik Schnetter: First, I defined schedule groups that do particular things at the necessary times there's an initial group, which is scheduled that initial.
Erik Schnetter: there's this post that group which is one post initial and post regret that's very brief story will apply boundary condition it's also running O D so was posted that's corresponding to more poster and there's the right hand side group which is.
Erik Schnetter: scheduled an odd service right hand side corresponds to the more the right hand side.
Erik Schnetter: Oh, and I said analysis group that grants at analysis I think I calculated the error and the energy there.
Erik Schnetter: And then the other things I just scheduled in the respective group their initial conditions synchronization boundary conditions after synchronization and oh.
Erik Schnetter: I don't know I hope you know about these things already, but each routine declares needs to declare which quit functions it accesses the reads and writes statement.
Erik Schnetter: I don't know how familiar, you are with it it's supported in the basic Einstein too good but it's not yet compulsory.
Erik Schnetter: At the moment it is compulsory to use these features in carpet X, so you have to add these annotations so this initialize routine writes the interior fi at side.
Erik Schnetter: And the driver actually keeps track of that and flags errors and synchronization it synchronizes the state group and of course synchronized expect these things to be valid interior and filter them because songs.
Erik Schnetter: And the boundary condition routine here that reads the state India to you, and why it's the stage on the boundary.
Erik Schnetter: and other things he Oh, he has an error estimator, this is the thing that actually.
Erik Schnetter: Estimates the local regretting error, so it reads the interior of the State vector and writes this read green arrow this estimate in the interior that's a variable in carpet X.
Erik Schnetter: And then of course the right hand side routine does exactly what you expect it reads fi and say everywhere and calculates the right hand side in the interior.
Erik Schnetter: scroll down a bit, and I have played synchronize the right hand side and apply boundary conditions to the right hand side, I think that just set to zero, so it doesn't read anything it just writes them.
Erik Schnetter: And at the very bottom of the file the energy which reads it everywhere energy depends on derivatives and writes the two year of the energy and synchronizes it.
Erik Schnetter: That means we don't have boundaries values for the energy anyway get probably not necessary and every calculus because I have an analytic solution out so bunker calculate the Iraq just for cross checking.
Erik Schnetter: So it reads the state vector everywhere and fights these fighters everywhere, yes.
Steven R. Brandt: there's a question in the in the chat okay um let's see where did you go okay.
Erik Schnetter: I opened a check.
Erik Schnetter: After closest needed for can be derived from reads and writes.
Erik Schnetter: And the after closest are needed, they cannot get be derived from reads and writes the schedule is managed in the flesh.
Erik Schnetter: So this is not handled by the driver, so the driver doesn't actually derive any of these after statements, the driver only detects in consistencies but it's kind of a good first step, but it's obvious that you'd want to address well.
Erik Schnetter: Let me go back to the presentation.
Erik Schnetter: So these dependencies in the schedule that was actually.
Erik Schnetter: The broken car protects was just piggybacking on other work that happened in the flesh and in other places before by other people I just when I began to develop carpet X I confuse myself.
Erik Schnetter: I thought it was important, I felt it would be a nice thing to have but I realized it's absolutely necessary, because I.
Erik Schnetter: mean with all the indexing self centered effects of metrics and that not understanding, but i'm access routing and new code I got so many sackfuls and undefined values it just out of self Defense.
Erik Schnetter: That I implemented all the scheduling clauses and I made it almost watertight so.
Erik Schnetter: That there's a lot of checking going on, so the basic idea is and you should know that from other discussions and the flesh.
Erik Schnetter: That you've scheduled function needs to declare which parts of which variables it reads or writes.
Erik Schnetter: And of course the grading synchronization prolongation and so on, they are handled by the driver the driver notes notes, which is red and what it written and it can also do this checking.
Erik Schnetter: out the driver checks that consistency and flags error, so it keeps internally track of which part of what variable is defined and if some undefined part is needed.
Erik Schnetter: And that would be an arrow but that's not the only thing dance, because the driver also checks whether the declarations is correct.
Erik Schnetter: First, of course, if grid function if scheduled function say does not access a particular variable variable isn't declared in the source code.
Erik Schnetter: that's not something carpet access that's also done by profession, implemented by other people so that's that if.
Erik Schnetter: function good function is only read and not written it's declared concept so there's an error message from the compiler if something is undefined.
Erik Schnetter: Then the driver actually sets it to not to not a number and debug mode only because that's expensive.
Erik Schnetter: And that's if a phone access is it that that would be detected and after the function return everything that's defined is checked whether it is still not.
Erik Schnetter: it's supposed to be not none so if something is you forget to set a particular great point the driver can detect it, and it goes even one step further.
Erik Schnetter: It check sums the parents of the variables that fun scheduled function promise is not to touch.
Erik Schnetter: So if the function sense i'm setting the entire year and it accidentally sets the boundary point or it goes point as well the driver would detect that as well and report an error as well.
Erik Schnetter: So it's not quite watertight if there but it's very close there and the driver also keeps track of.
Erik Schnetter: Which scheduled routine on what level of what iteration changes the state, so if something's inconsistent you get an error message is probably 10 lines long.
Erik Schnetter: And that says Oh, I need the ghost stones of this function to be valid, but they are invalid.
Erik Schnetter: Because three iterations to go there was this regulating and regretting only set the Interior and self reflect the outer boundary to invalid.
Erik Schnetter: And that helps your device these things so it's still some sleuthing necessary if the scheduling isn't right, but most problems are actually detected and it's way way better than SEC fall so tracking down and initialize point, so we have things.
Erik Schnetter: So I like I like that about it helped me a lot, and this is also by it's still necessary to pep that.
Steven R. Brandt: you've got another question in the chat okay.
Erik Schnetter: i'm opening to check.
Erik Schnetter: Our interior and boundary points, this was always tricky to pin down.
Erik Schnetter: So a domain consists of three pieces there interior points boundary points and gulfs points and the whole domain setup is a bit simpler than what carpet and the flesh supports also because of what comes from.
Erik Schnetter: And REX so you define a domain and that defines the interior of the domain and then every grid has kind of a layer corona of what you call it.
Erik Schnetter: And they have ghost points edit and if there is another grid nearby it's a ghost point and if it's near the outer boundary and you don't have any super to condition, then it's called the boundary.
Erik Schnetter: And that's that and it's rather straightforward, in this case HALO Thank you it's called Hello.
Erik Schnetter: And of course there's like inconsistency is because.
Roland Haas: You could.
Erik Schnetter: Be in a corner and then it's an.
Erik Schnetter: outer boundary point, and also a ghost point and it's still a bit fishy what happens there, I think.
Erik Schnetter: If something needs to be valid.
Erik Schnetter: I don't recall exactly how.
Erik Schnetter: it's handled.
Erik Schnetter: But, but when you loop of a points, then you can loop say you loop over outer boundary points, with or without costs, so that is that is checked there, but for the validity, I think it just is a bit careful if.
Erik Schnetter: If it's both an outer point and it goes points synchronization would set it, but if you don't set it in a routine that's fine if you say you set the outer bond or points and the boundary and goes points you don't need to set them or something it yeah.
Erik Schnetter: yeah sorry.
Erik Schnetter: But so before we get into more detail, I want to show a parameter file before actually showing the code, I want to show you the parameter file for a plane wave, so let me switch over to that.
Erik Schnetter: It is this one here so.
Erik Schnetter: Okay, the thorns you need carpet X you there is no carpet X live for similar because that's just an rx and embraces activated automatically that simpler.
Erik Schnetter: Oh, he saw us I mentioned that's the new version of moral who knows whether they emerge in the future, and of course the actual matrix if you do run.
Erik Schnetter: Since some time ago, you can have variables and parameter files we start with the dollar scientists, just to keep things consistent.
Erik Schnetter: So this is an important parameter here precinct mode mixed era compact X requires that.
Erik Schnetter: This precinct mode is this calculus checking that, for historical reason it's called precinct because the basic idea was that cactus should be able to figure out by itself when to synchronize variables and we're not not there yet.
Erik Schnetter: But I don't recall what mixed arrow means, but it means if there's an inconsistency, you get an error.
Erik Schnetter: That, then, you define the number of cells emmerich's likes to be self centered so everything is done in number of cells.
Erik Schnetter: And if a vertices vertices sit at the edges of cells, so if your domain is not periodic you have one more vertices and edges So here we have 16 selves on the course quick.
Erik Schnetter: And the thing is self centered so it actually the greatest 16 cubed plus costumes but if it goes vertically centered, of course, will be 70 cubed plus the boundary sounds on the outside and how many do we have I don't know whether it's declared.
Erik Schnetter: There is no declaration oh yeah I think the default is one goes sims of the greatest probably 18 q3.
Erik Schnetter: A few other parameters that you can set but that's a simple example that's not there.
Erik Schnetter: The maximum number of levels liquid every oh was debugging yesterday okay I set it to one trigger the ERA 16 is a good choice and that's the error threshold and web just choose one, and that also depends on how you define what the errors.
Erik Schnetter: Generic Roman quota integrator with the CFL factor, the initial condition and output, can you see something here yeah I output, all the variable in this side of format.
Erik Schnetter: Okay, that is a rather simple parameter file let's look a bit and that's a bit more complicated from the QC zero case from the.
Erik Schnetter: From rosetta for see system that we have here that yeah bit longer as you see, well, it said, our power file so there's a bit of fluff in the beginning, the Python script that generates it.
Erik Schnetter: Yet so nothing interesting to see here again precinct mode.
Erik Schnetter: Oh, I switched off poisoning and so on, that makes it much faster it disables much of the checking it still checks the Meta data, but it doesn't check the individual points, obviously, that would be expensive that checking in so on.
Erik Schnetter: This defines the domain X men and X Max chooses the domain size, it just tells me about it, and I think here is minus 128 to 128.
Erik Schnetter: And this is the size of the domain, that is, the lower boundary of the first style to the higher up a boundary of the last cell excluding boundaries it's just the domain and then we have, I think 256 cells so each cell has a size of one it's a clean thing.
Erik Schnetter: Then we can define a tire size i'm going to describe that a bit later this has to do with the open and people that ization it's not more efficient than carpet X and then carpet.
Erik Schnetter: And it's also used to em REX essentially each local grid that used to be called component and carpet.
Erik Schnetter: is split into tiles of that size so that's a large number essentially not splitting the X direction but split in the other direction, and each open MP thread works on one of these tiles at a time that is a.
Erik Schnetter: better way to split things on to open and P threats to make it more efficient.
Erik Schnetter: Then for historic reasons, a priority boundaries, I actually default, so I switch them off here and, apparently, you need this for statements to switch them off their rather like to be on.
Erik Schnetter: elder don't have any reflection boundaries and about just become outer boundaries, they go sizes sweet that also sets the boundaries is there's no distinction here, I think, in most cases that's not necessary some reading parameters.
Erik Schnetter: Ever estimator.
Erik Schnetter: yeah I don't know how to have a good master so it's these fake estimator that justice, the same thing as carpet Gregory to.
Erik Schnetter: You just put cube Thank you sonny sets the quick functions correspondingly I think it just uses one what are full of that gives you something similar attributes with cubes.
Erik Schnetter: And if the cubes get really down to the corners get cut off your choose the promulgation type dds is the good time that I mentioned that is interpolating for electric Center and leveraging for self centered the other ones as well or five request because sounds.
Erik Schnetter: And then the usual thing is just.
Erik Schnetter: To punctures and everything for performance reasons, you can switch off calculating the ATM variables are the right hand side of the stuff that is stored in ADM base and calculated the constraints here it's all on because I actually want that for debugging.
Erik Schnetter: Some data for see parameters and some output parameters, give us too much here sorry we put the output parameters on the screen.
Erik Schnetter: mode and P me means every dead many processes output, so you can say that not every mpi process outputs but only every 10 or 100.
Erik Schnetter: And the other processes would send their data to the nearest io process, using the fast infinity been that growth and that process then dumps the data to disk.
Erik Schnetter: Whether that's faster or slower depends on how fast the network, you have fun tell faster file system is.
Erik Schnetter: So here i'm running on our local system, and we have a GPS file system and it can be easily handled all the output so it's fastest if every process outputs, but if you need to buy to NF s or if you write down 10,000 processes i'm sure they will come in handy.
Erik Schnetter: output file oh yeah and REX has its own output format called flat file that generates a lot of files, I actually ran out if this quote or not because of space because.
Erik Schnetter: It generated a million output files when I wanted to make a movie I don't know it's perturbation per level per variable progress or something and get the enforcers.
Erik Schnetter: Know happiness so i'm going to using that anymore there's this silo format that comes from visit it's well documented.
Erik Schnetter: It starts to quit structure and the individual variables based on hd five layout is different than once, and their output some variables, the ATM buys variables, the wild scholars and something else here oh yeah I know that.
Erik Schnetter: Some of our constraints, the rms of all the constraints to check the evolution.
Erik Schnetter: asked the output it's not called tsp tab separated values taps about our values are.
Erik Schnetter: A generalization or diversion of csv comma separated values and csv is there a generic file format for data.
Erik Schnetter: So, if anyone is doing any kind of data, science, you get any kind of data, I mean you either need to scrape them out of an html page or they come in csv format.
Erik Schnetter: And anything can read that I mean Python Julia can do matlab mathematica anything can read that, so this is now our file format and it just puts separates the value by top signal block can also still waited and timer report on.
Erik Schnetter: Anything else I wanted to mention.
Erik Schnetter: Oh yes, there's a few other things I wanted to mention I didn't find them in here so let's go back to the slide.
Erik Schnetter: Oh, should I described that later yes i'll describe that i'll go to the next slide so I mentioned tiling earlier.
Erik Schnetter: This is the new opening people at a section these pictures are taken from the emmerich's documentation.
Erik Schnetter: And here, you have some costs could invite and find great with to grits on it to components conflict would call it.
Erik Schnetter: And they have a particular size and in carpet be open and people that I section would work as follows it loops serially over these two components.
Erik Schnetter: And then each of these components would be broken up into individual pieces by opening up but there's still the serial over the components.
Erik Schnetter: And in carpet X is more efficient, it shows that a particular title size here for squared and breaks all the grits up and then you end up with eight titles and all these eight titles.
Erik Schnetter: open and be paralyzed at the same time, so the advantage is more trials there's more parallelism is more efficient.
Erik Schnetter: And the disadvantage is that a scheduled routine runs only over a single time well that's a disadvantage The disadvantage is that this is different than the way it used to be before so you need to adapt to doing that.
Erik Schnetter: And mentioned the recruiting before that I think you cannot have arbitrary refined regions but rather it's the dolls it done in blocks of eight and also there is a Max criticized in an rx if.
Erik Schnetter: it's a carpet carpet does if there's an arch good it splits it into pieces to ensure it can be evenly distributed firewall processes and REX takes a different approach.
Erik Schnetter: It just takes all the grits and breaks them up into well grits that are, by default, I think 32 cute.
Erik Schnetter: And these are then open it up by sorry mpi privatized so then grids and tiles grits are these larger things and you specify the grid size and these grids have been distributed.
Erik Schnetter: And the grids are broken into tiles and the titans of industry muted so various carpet looks at the number of process and threats and tries to find an ideal distribution for that.
Erik Schnetter: Corporate access it the other way around that's just the maximum size and then you get chunks and bits and visa distributed and if it's not efficient during the tissues smaller grits or smaller time.
Erik Schnetter: Now the advantage of all of that now just because i'm so proud i'm actually going to mention that, as I ran some benchmarks with ios tales and grid sizes and so on, on a single note of a particular benchmark that's a.
Erik Schnetter: mean kosky on 256 cute and for 10 innovation, the best thing I got was 57 seconds it gets lower as you go down here.
Erik Schnetter: And the interesting thing year is the number of so the title sizes for back to and so on, and the interesting thing is the number of threads process.
Erik Schnetter: This one year, so the most efficient way of running this particular benchmarking carpet X is using 10 open and P threats or mpi process.
Erik Schnetter: And that was always different in carpet carpet, it was always the case that NPI is parallelization is more efficient than opening p.
Erik Schnetter: People would not use open ap only if you hit the hit the scaling limits, when you run on 1000 course that's when you switch on open it up.
Erik Schnetter: But here we can use open up in the beginning, so that alone means that particular benchmark holds water that things should be 10 times more scalable in terms of using number, of course, and that that i'm quite happy about that.
Erik Schnetter: and
Erik Schnetter: Yet I don't know whether I should show this slide is kind of near and dear to my heart, but I also don't want to override probably a bit of time of questions.
Steven R. Brandt: We have another we have another question in chat.
Okay.
Erik Schnetter: This is due to the cash locality, of having small tile grits to loop over, I guess, let me go back to tiling so each child needs to be so, if you need to have.
Erik Schnetter: Not just a local variable for a great point but kind of a local grid function, or something, then it's important that you have tiles that are small enough that they fit into the cash.
Erik Schnetter: And the way things happen in the data for see implementation is that first all the derivatives are pre calculated and then the actual right hand side is evaluated.
Erik Schnetter: And that obviously means that after calculating the derivatives, they should stay in the cash until they needed by the right hand side, so you need tiles that are small enough.
Erik Schnetter: And because cache lines are 64 Bytes and as simply vector ization you can do in this direction it's most efficient to not tie the X direction, but in the.
Erik Schnetter: y and zed direction you apparently four by two or two by fours the most efficient one two by two is also good it's not much slower and what is that.
Erik Schnetter: percent or something 1% slower or 8422 variables it's good to have a set of mercy and, yes, that is probably cash locality, because you kind of local variables that spend a whole time.
Erik Schnetter: is also important for hydro, when you first calculate fluxus and then take divergence of the fluxus you don't want to have a FLEX grid function that you feel, because that would be really large and you need to write it back to my Mare and read it again.
Erik Schnetter: But this setup you can have a local FLEX grid function that is only for one time that is actually more efficient.
Erik Schnetter: Well, I owe some smaller differences, first we learned that.
Erik Schnetter: modifying files after they are written is generally a bad idea, because hbc systems are unreliable if you open an html5 file for writing.
Erik Schnetter: Then, then, then something happens you run out of time, the out of disk space SEC folder something you lose the whole file was actually an unsafe thing to do.
Erik Schnetter: So the basic idea here is that each time step data gets written into one file and the next time step into the next file.
Erik Schnetter: And also betray true but gives us few files as possible because it's more efficient, I mean you can open 100 fights but you can't have a 10,000 files directory.
Erik Schnetter: That would not be efficient norms got rolled into one file to see I mentioned it earlier that this, then you ask a format that just call it to speak.
Erik Schnetter: Is one file for a group when it's ready to buy everything you can open one of these out ascii output files into excel that's actually quite beautiful.
Erik Schnetter: And I don't know when I was younger I hated excel but it's quite Nice because you see the numbers there they're nicely formatted your market column that, and you can do something there, I mean I would use Julia Python, but you can do it, so why not why you're not use a standard format.
Erik Schnetter: And our new html5 format is based on silo this standard visit format.
Erik Schnetter: And there's a reader and the writer fridge but not yet checkpoint and recovery, the Meta data correctly set up.
Erik Schnetter: And it would be straightforward to add a compatibility layer to recreate the old carpet forward.
Erik Schnetter: And you would just create one additional file that contains all the metadata that the old format requires and then you add external links to the other file that has the actual data.
Erik Schnetter: And the hdr five utility utilities to to combine files and to split files and to expand links, so you plays a link by actually copy of that object if you don't like external links and so on, this can can all be done.
Erik Schnetter: Okay now oh yes let's go to the actual source code, I could carried away a bit slow in my presentation here.
Erik Schnetter: Luckily it's not that complicated.
cpu.
Erik Schnetter: A few things that are different.
Erik Schnetter: briefly go over them you probably already familiar with this one arguments checked that is one of the things that enables the undefined and cons variables.
Erik Schnetter: Depending on what you declare in the scheduler highly recommend that that's the line.
Erik Schnetter: And then, when you declare variable sorry scroll down to the initial condition here you write declare arguments and then the name of the scheduled function is that of the character minds highly recommend that, of course.
Erik Schnetter: And then the way to loop of a brick functions.
Erik Schnetter: So it turns out that if you have vertically centered and salsify good functions, they have different index bounce and you need to use a different.
Erik Schnetter: A different g of index speedy needs to be slightly different and so on, and once you have the result of God index vd is just a single integer it's very difficult to to index checking.
Erik Schnetter: So there is now a c++ rapper class that of course is very efficient, but which allows you to do index checking into that mode.
Erik Schnetter: Unfortunately, you have to create objects of this thing I mean essentially just wraps the pointer, but you have to do it yourself there's a bit of point upload.
Erik Schnetter: So here we take the value file and create this rapper it's a called G F five I don't care about five down here, and this weapon, should be the auto generated and Jeff I is now this thing that behaves like a 3D array and fortran index checking in everything.
Erik Schnetter: And there is this layout class which defines the layout with this index type here, so this is now the layout for a self centered variable.
Erik Schnetter: Everything sell some that i'm only needing this layout here, and then, if you do something, have a loop, then you set this variable at the index I.
Erik Schnetter: So people are used to writing a J K every time but there's really no reason to write a jk because the current grid point is always I J K, so I use a multi index with the big guy uppercase I, which corresponds to I J K, am I sending it to I don't know standing wave and so on.
Erik Schnetter: And this is then you will looping macro.
Erik Schnetter: Also written in c++ so you look over the interior, this is the index type because you only want to look over the into your fat corresponds to self centered function.
Erik Schnetter: And then the user c++ Lambda and then point desk is kind of the current point that is executed here, and if you do it like that it's also a and easier to.
Erik Schnetter: convert that into cooler code at this point description P contains this mantra index year corresponding to the current good point and it also coincides X, Y zed if you need that required coordinates I need to calculate that manually.
Erik Schnetter: And I might be going too fast, because I feel pressed for time, so if they have questions here.
Erik Schnetter: But I guess you just see that this is probably rather easy to read, and you could copy that is called on a bridge to some other function.
Erik Schnetter: As you have the boundary function, the setup here is the same thing.
Erik Schnetter: But then you have loop bound loop of the boundary of the thing and that of course is internally six different loops or whatever, depending on the ghost towns and so on, because it looks of all the different interfaces that outer boundaries.
W. Kastaun: The investment that say auto plant in that same vein to have admitting operators abiding on hold that functions like you can multiply let's say two grid functions and put in, so what it is.
Erik Schnetter: I did not plan doing that it would be straightforward actually i'm REX itself supports that.
Erik Schnetter: So you could call it a function to get the underlying emmerich's array for a grid function here and could do it like that, but we could also add the respective operators and hold quick functions that would.
Erik Schnetter: Probably 20 lines of code or something would be possible The thing is, if you have a sequence of operations, then it's important that you use all the loops and do it and and go.
Erik Schnetter: So if you calculate two X plus one you don't want the two X two kinds of friction and temporary object and then you look over that again in the one.
Erik Schnetter: And with that if you buy the indices yourself then it's easier to ensure the fusion, but if you find the other thing easier to understand for X be fluxus shift to left shift right or something that would also be possible.
W. Kastaun: suggesting is that we have sex passion template or something, but, for example, for initial data performance doesn't matter so epic easier simpler code is.
W. Kastaun: Also, a good argument to have something next.
Erik Schnetter: Yes, yes, no don't we could we could do that.
Erik Schnetter: template of right and that.
Steven R. Brandt: i'm wondering Eric I don't know where you are in your presentation i'm wondering if we want to have a second.
Steven R. Brandt: present I mean a continuation of this at another call do you want to try to do that, or do you Okay, I did not sell to you.
Erik Schnetter: This is the last thing i'm presenting after that I have my final slide.
Erik Schnetter: But, since this is the first time we actually looking at the source code I don't want to rush it either.
Erik Schnetter: i'd be happy to go over time, or to continue the presentation at some other time, as people were.
Erik Schnetter: That, let me continue, I want to show you the right hand side here, and of course in the right hand side you have fi and sigh and they are cons here because.
Erik Schnetter: You have two together and cons because you're not supposed to write to these that's how the function declared it.
Erik Schnetter: The right hand side variables are not cons, so the outer cons is just for the rapper object for the pointer, so to say that point has cons but, that being pointed to is not cons and then you loop over the Interior and then you have this.
Erik Schnetter: You have this back here, oh no because they should totally be zero here, thank you for pointing that out, so this is the current.
Erik Schnetter: index minus di in the zero direction in the X direction in the one direction in this direction, there is another syntax where you can write I J K, if you prefer, that as well, I mean you could write a J K here and then you're right I minus one jk jk I.
Erik Schnetter: i'm not quite sure, but it's easier to understand, I prefer that syntax and more complicated expressions, but I our stencils, but when I show it to you see it for the first time it looks at the helm of me.
Erik Schnetter: And you know you've seen loops there's loop in loop boundary that's probably also a loop all down here when you calculate the revenue calculate the error disappear you do for all points and the error is that the current stage minus the standing wave at that particular time.
Erik Schnetter: Okay.
Erik Schnetter: Okay, in the interest of time, I shall switch back to the presentation.
Erik Schnetter: and
Erik Schnetter: The last slide so what I think the immediate next step, are in the project first thing is to help others get started.
Erik Schnetter: And the big changes that people are facing are probably the schedule classes that are necessary.
Erik Schnetter: The fact that there's no sub cycling's area of a complicated thing that wants to do special things with adaptive venture funded you'd have to stop set.
Erik Schnetter: And the loop tiling which requires that you loop over only a single title and your scheduled function not over everything the loop back to that course that correctly anyway.
Erik Schnetter: If you are writing your own loops, then you need to look at the title size and not that lsat.
Erik Schnetter: And big projects implement checkpoint restart that's the last big feature missing before its production ready.
Erik Schnetter: Obviously, want to test performance running on 100,000 cores to see whether we can reap the promise benefits and then do some physics by there.
Erik Schnetter: And without subtitling in time it's probably difficult to get something interesting with say hi spinning black holes, because you need a lot of refinement levels that couldn't be efficient, so maybe something whether whatever it is the microphone sounds good.
Erik Schnetter: That concludes my presentation questions.
Steven R. Brandt: Thank you Eric oh that was that was great.
Steven R. Brandt: i'm wondering about you were showing that code just a minute ago with the index type and all that stuff that you were declaring i'm wondering if that couldn't be generated, for you know, in the declare seat seat decay arguments macro it's.
Erik Schnetter: It totally should yes.
Erik Schnetter: One problem there is that you notice I call the Jeffrey to.
Erik Schnetter: there's a freebie Jeff really to i'm actually arrived at g of 55 in the meantime, because that that that design wasn't quite right for kuda and so on, but I think once we settle to something that is actually nice it should totally be auto generated for c++ code.
Steven R. Brandt: Okay, do you, so this is seems to be relying on a lot of c++ features, what do you think is going to happen with fortran support and like.
Erik Schnetter: I think 14 support could rather.
Erik Schnetter: easily be added, most of the c++ support is necessary because c++ doesn't have 3D erase buildings, you need to do a lot of stuff yourself.
Erik Schnetter: If you're fortran support you need to make sure there's an easy way to get the bounds of about.
Erik Schnetter: Something should be iterated so if there is a function, the function is declared over.
Erik Schnetter: say to iterate over self centered away, then you need to pass in obviously that the size of the way, but also the size of the title over which it should iterate and then you use the regular fortran indexing then that's all fine so.
Erik Schnetter: Like c++.
Erik Schnetter: But I see it, straightforward to keep that am REG cells that has fortran support so it's not difficult.
Steven R. Brandt: Okay i'll always a asks if hydro can.
Steven R. Brandt: shoot it just scrolled away from me.
Steven R. Brandt: If if hydro can easily be generated to the hydro toy do multi fluid or can it be easily general generalize to multi fluid.
Erik Schnetter: So the hydro toe is non relativistic it is really just a basic thing is that everything is linear is easy as possible, the only thing I experimented with there was.
Erik Schnetter: The optimization the loop tiling whether that would give us benefit, and yes, yes, it does, and it also, of course, we flexing and so everything everything that works so it's just testing the amex features there's no physical and they're.
Erik Schnetter: easily extending guess sure you just add a few more good functions and so on, and put all broke, but really everything is quite primitive in there.
Erik Schnetter: And he asks that are the advantages of carpet X over pure mx the advantage is that.
Erik Schnetter: Car protects comes with the Einstein tool kit and there you have additional things initial data arise and find us and our analysis modules.
Erik Schnetter: If you're quite independent, then you can write your own and REX based code, you have to, of course, also do some mesh refinement that regarding sub cycling and so on, yourself, your time integrator but it's certainly feasible.
W. Kastaun: find that would also have to.
W. Kastaun: be adjusted to the new framework right, so I cannot just combine an air mannix simulation with counter eyes and find out that users plan carpet Oh, can I.
Erik Schnetter: Add to my knowledge, existing horizon Finder should work I haven't tested it bill which probably means it doesn't work, but that does not not nothing big missing.
Erik Schnetter: The rice and find our only needs interpolation and interpolation is supported and thanks to the work off of Poland helped me who who did that.
Erik Schnetter: Thank you, even supports that correct interface the interface that you're trying to direct expects.
W. Kastaun: To hold for scale for the composition and stuff like that that relies on the on the interpolation.
Erik Schnetter: I don't know the regular widescale a calculation, does it for every good point so you need to update the loops there.
Erik Schnetter: And, but for wave extraction, we can just extract on a sphere and do something there that should also work.
Roland Haas: So word of caution, so a lot of some of this as planned, I think, to support for the interpolated Sam has added is probably missing bits I don't think you can quite get the horizon well Maybe you can get present to work because it relies on derivatives and God knows what else.
That side.
yellow if you have to hook up.
Erik Schnetter: I think the hook up isn't it but.
Roland Haas: It might be, there was a bit of a Cluj but in principle the idea is that that is one or two things that should be supported to just take over because it doesn't depend on the grid.
Roland Haas: give it a try see what happens.
Erik Schnetter: i'm going to continue with the comments Gabriella says.
Erik Schnetter: Everything I know about the I said to Chris because there are examples to have a lot of documentation and examples, yes.
Erik Schnetter: So at just this wave to cpu found actually realized yesterday there was no simple example, so I took one of the existing websites simplified it called it vector cpu and just pushed into the repository this morning.
Erik Schnetter: That was the one way to fix the bug derivatives wrong so clearly knitted with a needle but yes, that is there and I also hope.
Erik Schnetter: That, as others begin to use it, others help us with these testing and with writing examples and documenting examples and so on.
Erik Schnetter: insects cactus provides higher level abstractions then arm regs definitely cactus has all these interface CCL you say this is my state vector.
Erik Schnetter: These are my right hand side variables, this is the boundary condition function and then.
Erik Schnetter: You say, well, let me call this particular function that expects these parameters and then I restrict from course to find grid.
Erik Schnetter: And all the checking of quick functions and consistencies and so on, nothing of that is there, so essentially.
Erik Schnetter: i'd say half the work that we did to write copy decks you'd have to redo a couple of decks a generic so maybe you don't care about hydro, you only want to play call you can skip all the work on self centered functions, but.
Erik Schnetter: If you want to have a real full resolution code then you'd have to be do a lot of things they did it with carpet X.
Erik Schnetter: Multiple skill for what's the highest find a difference in order attempted to set a fussy and carpet X litigation order.
Erik Schnetter: I believe you have generics stencils and the stencils are just essentially a table I don't know up to what order we ah, maybe 11 I like it if things go up to 11.
Erik Schnetter: is better for see currently runs with fourth order but that's just been defined in the beginning.
Erik Schnetter: And oh yeah I don't think the eighth or stencils and the code, but that would be again just three or four lines, I mean you need to first derivative second derivative and respective dissipation operators up winning operate okay it's 10 times that you need to get noticed.
Erik Schnetter: And the publication restriction order that's what provided by carpet X, they are quite high older I don't recall, where the limit is, but you can also extend that up there.
Erik Schnetter: that's a mix of Python bindings to postpone speedy am our data.
Erik Schnetter: I don't know what you mean so this file format can be read.
Erik Schnetter: If that's what you mean.
Erik Schnetter: But you probably want to have a kind of a connection to Mumbai, you have the number stuff in Python and then REX does the calculations there, and of course number has a race and outbreaks as a whole grid hierarchy, that would be cool.
W. Kastaun: we've been doing, I don't know if m access things like production operator so interpolation operators, one could even if it's just to see a people could call it homeless and patent.
W. Kastaun: That would probably simplify biting post processing libraries or not, because that was awful thing and carpets that one had to reply to the whole Mr loading in tight and again, which is not an option okay.
Erik Schnetter: I completely agree this is really important, we made some half hearted attempts well I don't think it was half hearted, but we made some attempts, but eventually we failed because it was too difficult.
Erik Schnetter: I think right now that carpet X, is still in a young stage we should try to do that I don't know whether i'm REX is the right point because we have higher low high level information.
Erik Schnetter: The way we use emmerich's basically and regs only knows about refinement devils and operators between the refinement devils we don't actually ever tell me about the whole grid hierarchy.
Erik Schnetter: Because if we do that, then does no good ways to shoehorn our own time evolution mechanism in there and so maybe carpet X should have respective things.
Erik Schnetter: I wonder how that would work, maybe we need to start it up as a server and then it takes requests louisa was working on that and the connection with visit to have a visit back and engine running there and that in the end was not a very good idea, and I think.
Erik Schnetter: This communication with the visit designers that they were not actually actively working on it.
Eloisa Bentivegna (IBM): Sorry, under this side, though, is just not not developed too much as a proof of concept that the.
Eloisa Bentivegna (IBM): function.
Erik Schnetter: Okay well.
W. Kastaun: That would be our sentence and movie industry developed this open DDB library, which is also unstructured good.
W. Kastaun: And value, you have really nice Python bindings actually use that in a tracing.
W. Kastaun: Practice data.
W. Kastaun: So if carpet excellent be sort of a library, that is called in I tend to get and can also be.
W. Kastaun: called from outside.
Erik Schnetter: Yes, yes, the problem is that characters as a framework, and he likes.
Erik Schnetter: To be in control of things so calling carpet acts like a library because carpet X depends on the flesh on calculus for various things on capital really wants to be a framework.
Erik Schnetter: I think the easiest thing might be you started cactus thingy in the end, but it doesn't do time evolution, but you just tell it to read some great functions get a good structure there.
Erik Schnetter: And then, it just sits in a tight loop waiting for requests from the outside, and you have kind of a generic analysis on that fights for a request from Python and bed duster respective operation adding subtracting producing interpolating that should be possible.
Erik Schnetter: And then, when you want to load a new grid structure you'd have to restart that service so it's not ideal, but what I just described, would be relatively straightforward to implement.
Erik Schnetter: Someone taking notes, this is a really good idea we should put that we should put that on a to do item at least explore it further and and see see why it doesn't work, but I think it should.
Steven R. Brandt: we're recording so that counts.
Okay.
Erik Schnetter: something about kudos and accelerators yes i'm ready supports accelerators there's progress every month there's of course cooter.
Erik Schnetter: There are other things there's open cl calls and other frameworks that are being developed it's not yet quite real.
Erik Schnetter: Whether there is a framework that works everywhere, the moment i'm sad that my MAC laptop doesn't support cooler because it doesn't have an nvidia graphics card, it has a graphics card, but I can't use it.
Erik Schnetter: And so that mentioned briefly earlier what we did in the past week we looked at accelerator support.
Erik Schnetter: And there is another phone that looks very similar to backtrack cooler a few small changes it still runs basically with the same efficiency.
Erik Schnetter: On a cpu, but it also runs on gpu in the sense it runs a reasonable their gifts correct ourselves the performance probably isn't ideal, because if you want to brand performance on a gpu you need to have different tiling sizes different sizes.
Erik Schnetter: And most importantly, you need to make sure that all the calculations had and happen on a gpu that including same conversation and prolongation.
Erik Schnetter: And those stencils the ones that carpet X uses all executed on the cpu there is done a lot of copying forth and back, I think, even a time integrator probably wants to do stuff on cpu.
Erik Schnetter: But that can all be added, but as a proof of concept it works, we can run the Einstein equations on gpu but performance is not there yet, and I.
Erik Schnetter: am happy that the same code runs efficiently and cpu and reasonably on a gpu So if you write the code and follow the usual coding guidelines.
Erik Schnetter: And you would have a code that should run on a gpu kind of US ugly but not in production out of the box, and that is actually something pretty cool.
Erik Schnetter: circle circle, yes, yes I looked at the sequel PepsiCo and so on, and I don't call many of these things work beautifully on Linux but then I want to compile it backwards.
Erik Schnetter: So haven't found a good thing to get.
Erik Schnetter: A repository for carpet X, everything is the same repository and wrote on to put the URL into the email announcement it's on bitbucket somewhere, but let me just posted it yet until the warden beat me to it.
Erik Schnetter: Oh it's not called carpet X is called cactus and REX that makes it difficult to find decks the URL everything's public up until a week ago there was a big coup de branch that's now been merged that's what I mentioned, so the.
Erik Schnetter: Other development happens on the main branch master branch.
W. Kastaun: office carpets sometimes for needs.
W. Kastaun: I can tell that's because that's lacking functionality, for example, i'm making histograms and I use a carpet vate you say a similar concept in am next this debates of.
W. Kastaun: find that volume sets.
W. Kastaun: If I want to integrate or was it something that is not just the grid function.
Erik Schnetter: So there are weights and, obviously, if we use a cell Center thing the way it is just the cell volume sounds so easy to integrate.
Erik Schnetter: And for vertices they are also weights I don't know whether they're stored are calculated.
W. Kastaun: Does appoint being not counting the same thing it's Nice because it's once in a class in once on a final.
Erik Schnetter: Yes, yes, yes, of course, good points, whatever rate of zero If so, find great point.
Erik Schnetter: And at the boundary between the course and find grid, you want to count the course but by one half and defined.
Erik Schnetter: By one half of one kind of the Left contribution to pass the course, but the other one the right contribution towards the fact that so that thing is there as well, I don't know whether.
Erik Schnetter: it's a great function that can be output.
Erik Schnetter: Okay, is there a getting started tutorial yes, there is it's been recorded as soon as this recording stops you have your tutorial nope i'm sorry.
Erik Schnetter: This is the first time, be starting with tutorials like that, before that it was really quite technical, but if somebody wants to help us make a tutorial that would be much appreciated.
Erik Schnetter: Now, and this talk i'm mainly the we mainly discussed how to do things in an abstract but it's not hands on.
Erik Schnetter: One big problem obviously is that you have to compile an rx yourself if you want to as an external library.
Erik Schnetter: I don't think that happened automatically it so we thought about at some point, having a tutorial that is more hands on.
Erik Schnetter: Where people write their own code and then the real questions come up and and their things and took it out put.
Erik Schnetter: Where we can also discuss that so i'd be happy to schedule that at some other point, it could be this time slot or, if you want to have more time could be another template that has low on the mailing list.
Erik Schnetter: of other questions comments.
W. Kastaun: So you mentioned this Melinda placement is set aside money not molded is that I am a carpet X symbolizes also for mccullough carpets and if, yes, is there some description somewhere.
Erik Schnetter: So this thing is called OD solvers it's in the same repository.
Erik Schnetter: Is the carpet X arrangement and, at the moment it's tied to carpet X.
Erik Schnetter: And to amex because it wants to allocate temporary grid functions that was always a bit unclean in mobile type levels and allocating something.
Erik Schnetter: In here you just asked for temporary storage for a good functions and release is that afterwards.
Erik Schnetter: and that of course makes everything simple also the loop looping doesn't handle it have an inner schedule anymore it happens in c++ code, you can call schedule groups from from secret carpet, so a cactus supports that.
Erik Schnetter: And that means the whole code is much simpler and I think there's even the generic thing there that handles opportunity butcher tableaus and then you.
Erik Schnetter: Put your numbers in there and then you get an eighth order rumor kota.
Erik Schnetter: it's not quite efficient if your hand coded, then you can release some storage of them and needed right hand side you earlier, at the moment it doesn't do that yet, but that's that.
Erik Schnetter: And there's also at some point we wanted to look at the imax integrators implicit explicit so a few methods are implemented there it's not difficult you just put your clothes and so on.
Erik Schnetter: But I don't think we actually properly tested them particular could purchase test whether things are corrective second or third order I know don't want to say that i'm sure well I tested them he's always very careful about that we run a test.
Roland Haas: yeah we had to have convergence just actually the soft tests okay.
Erik Schnetter: And so what I would have liked is a convergence test with an analytic solution and it wasn't able to construct, something I mean very kind of predict what the next thing that's actually so that doesn't exist, but a conversion system in the sense of self convergences Lisa in there.
Roland Haas: We know I mentored and test i'm sorry.
Roland Haas: Pardon me I didn't test i'm sorry only tested the audience over itself.
Okay.
Erik Schnetter: It should be the same kind of thing right you just have to write insights implicit and explicit it should converge, the same thing.
Roland Haas: But it's the time when i've tested yes.
Roland Haas: yeah okay.
Erik Schnetter: So back porting this O D servers to.
Erik Schnetter: carpet.
Erik Schnetter: It would be difficult because carpet doesn't have the idea of good functions on the fly it wouldn't be difficult to add, but if it's not.
Erik Schnetter: For questions.
Erik Schnetter: Okay, and in this case I apologies going over time, I should have anticipated these kind of questions and kind of shorten my presentation.
Erik Schnetter: I tried to be quite explicit about the features that are not yet supported because I don't want to fool you and I also wanted to have slides where I described the details of where things change between corporate and corporate X and that was a bit more material than I anticipated.
Steven R. Brandt: I think it's great I was expecting it to go over I don't know what other people are.
Steven R. Brandt: All right, well, thank you, thank you Eric.
Steven R. Brandt: Ah, for this.
Steven R. Brandt: This is this was really great.
Erik Schnetter: yeah muskoka yes.
William Gabella: Yes, it was Thank you.
Peter Diener: Okay, so i'll stop recording yep.
Miguel Gracia: Thank you.
Steven R. Brandt: See you all.