AI Assistant
Transcript
00:00:00Today we start talking about different language,
00:00:03which is closure where coming from Ireland.
00:00:07This is that it is the functional language,
00:00:11but while for our language.
00:00:13This is very much on the functional nature of
00:00:16the language and we just in time that the facts that it might
00:00:20be good for working with currency rather their we beat about
00:00:26support for distribution for
00:00:28tolerance and the actor model in general year with closure.
00:00:32We will try to insiste a little bit more
00:00:35on this idea that if you have
00:00:37functional language for the art and
00:00:40the evaluation process by copy rather than by side effects.
00:00:44You can naturally transform
00:00:47program into the current version of the same program
00:00:51because independent expressions can be evaluate in
00:00:54any order and also
00:00:56currently without working about side effects and.
00:01:00State of the staff might cause
00:01:04some ages so we will be doing something in action
00:01:10with closure closure is not a strict as long for
00:01:16being pure it has also some non pure
00:01:20expect that we have this not so in the title.
00:01:24IT has the possibility of having
00:01:27shared states between the different threads.
00:01:32This shared states can change you
00:01:36have to be state principle might be problems,
00:01:39but it takes a solution,
00:01:42which is my view to have a look at which
00:01:45is win for my table shared state in
00:01:50a very limited part of the program essentials that data structures
00:01:55are immutabile e un bot immutabile
00:01:58Is only references to this data structure,
00:02:01which can be shared so you have to be careful about
00:02:03it is providing also mechanism for
00:02:06support a safe use of
00:02:09this shared table references so this is more of
00:02:13the program and we start discuss the language as usual
00:02:18with the LG description from the designer developers inventor.
00:02:24The language is a functional programming language and it is
00:02:27created in one of the languages,
00:02:30which are still in used, which is lisp.
00:02:33I don't know if you make it is
00:02:37very compact language beautiful hands on the point of view.
00:02:43IT is very old it is from the sixties.
00:02:46IT is the second of language, which is used.
00:02:50To the best of my understanding.
00:02:53The first is for Which is coming from
00:02:57likely before instead closure is not hold even if
00:03:01it is strong inspired to it is from to
00:03:05seven and it is started as is a personal project.
00:03:11Just for fun by
00:03:12later some interesting become funded project and
00:03:16now it is not as famous as the other languages that we discuss,
00:03:21but it is not english language.
00:03:25What we like one reason?
00:03:27Why we are discussion about this is
00:03:30that it has been designed explicit to be
00:03:32a current language with support for currency
00:03:36and as we mentioned before trying to leverage on the immutabile,
00:03:43which is distinct features of functional language,
00:03:48but it is not in the science and it.
00:03:51Pragmatica, pragmatica view for some forms of
00:03:55my stability and also choosing to be simbiotica with something,
00:04:02which is extremely wide spread,
00:04:03which is the JVM.
00:04:05IT is not compile time machine could be to
00:04:08the JVM and it is fully interoperabile with Java.
00:04:13But historical everything that is back to least which has
00:04:20been developed design induced by
00:04:24one of the giants in the history of modern computer science.
00:04:28John McCarthy, which is known as
00:04:31the uncle maybe because of the nice looking expect.
00:04:39IT was introduce when McCarthy
00:04:43was working to meet laboratories and the ideas as you
00:04:47see from the title of the people who that introduce
00:04:50this language Post develop
00:04:53a language level language
00:04:56capable of doing some forme simboliche manipulation of expressions,
00:05:01but still with the same expressive power at
00:05:07the calculus and so probably Turing complete and
00:05:12not efficient so retailing also
00:05:16the efficiency what you have with the machine code by now this
00:05:20looks like and you have to this is not
00:05:25very long time after Turing This
00:05:29is very early times of computer science and the reason.
00:05:33Why the language was introduce one of the main reasons
00:05:36was to do this simboliche manipulation and stuff,
00:05:40which intended to be used for reasoning so
00:05:44for what John McCarthy decided to artificial intelligence
00:05:49is the one of the artificial intelligence
00:05:53of artificial intelligence was the Turing white paper machine thing
00:05:59that is the artificial intelligence is
00:06:04to McCarthy and so scientific again.
00:06:10We're standing on the word of Giants.
00:06:14With the beautiful theory and solid rock theory behind,
00:06:19but as we the language also pragmatiche in
00:06:22several respect as it is not functional even
00:06:26if it right to exploits some aspects of
00:06:29the functional languages so you have some ability,
00:06:33but limited and it has also
00:06:37some control structures and this is maybe not the nice,
00:06:42but it is to the efficiency
00:06:45and Kind of overcome the take that you don't have
00:06:50tail call optimization and it is the JVM and so you can access to
00:06:57the library of Japan and
00:07:00as it is not may be wide spread as other language,
00:07:05but it is not just language and this is what we plan to do.
00:07:14We start usual with the basics.
00:07:17This will be short talking about
00:07:19expressions and types that we need for example
00:07:22and wireless it and we will
00:07:26discuss two things essential one is this idea that
00:07:31making a program current functional languages so that you
00:07:36can make it almost automatic and so on
00:07:40the infrastructure is supporting
00:07:43automatic functional currency or
00:07:45data concorrenza for free as it is called
00:07:48something and then you can also program yourself
00:07:52some form currency this we will see.
00:07:58Some relations which are there also many other languages,
00:08:02but see them enclosure
00:08:05about futures promises and software transaction.
00:08:11Basics. And we are
00:08:17the functional languages so we have to talk about expressions
00:08:21expressions evaluate may be one peculiare aspect to be
00:08:26mentioned is that as it is this is the
00:08:32same as is also may be as in the least com from
00:08:37essential there is not distinguished
00:08:40between could and that they are the same thing.
00:08:45Less everything is a least in least and also enclosure something,
00:08:50which is the language high level of activity said in heritage from
00:08:59the times and from
00:09:02lambda calculus and may be also from
00:09:04the early ideas of Turin because you
00:09:07remember that the idea of computer comes from the intuition that
00:09:15is not destination between data and progress and this
00:09:20is really taken seriously in and closure.
00:09:26Everything is progress data your tools
00:09:31for manipulation could essential
00:09:33by freezing the evaluation of code,
00:09:35which would be wise interpretate as
00:09:38an expression some manipulation later forcing
00:09:41evaluation something which makes the language easily
00:09:45accessible and if you then look
00:09:47at the documentation you will see that a lot of things,
00:09:50which looks like primitive common language are actually Macross,
00:09:54which are preprocessore and transform to core of basic contract,
00:10:01which is extremely limited and sometimes col lisp and
00:10:04languages in the family programma programming languages.
00:10:09We want say very much more about this.
00:10:16Basic things very basic things the values you can expect
00:10:20all kind of values you will
00:10:25be java strings and then you have
00:10:28keywords which are the same things.
00:10:33Erlang atom values switch are already in normal form,
00:10:40which are you see more for the same names so
00:10:42as keys for the maps or for
00:10:46tagging structures and like
00:10:49to start with the colon and the top ten,
00:10:52half string and you have least.
00:10:58Almost everything is a list
00:11:01in particular when you have the least you interpreti it is
00:11:04an expression where the first element is
00:11:06function and the tale of the least
00:11:10is the argument with the function takes and.
00:11:18This is a small example taking place of one two you
00:11:23can you can eat even this way at
00:11:28the end of God a lot of parentesi and some people say that
00:11:32is the name list should mean least process,
00:11:38but other people who doesn't like this simulation of 46 that least
00:11:44list stands for list of stupid parents that.
00:11:53You can associated names to pieces of closure data.
00:11:59So you can defined variables as inner here they are to as bars.
00:12:06This is the syntax you
00:12:10have the name of the bar and an expression here.
00:12:14IT is just a value expression and value what is associated to the.
00:12:21Expression can be a real expression that needs to be evaluate
00:12:27and after that you can think of the war as
00:12:30a reference this is reasonable.
00:12:34Trust full. Metafore Es.
00:12:39Reference to the value.
00:12:43The evaluation of producers
00:12:45the corrisponde value and the difference for
00:12:49this is that you can mutate the so you can change the binding of.
00:12:55VAR can be shared because their threads and so they can be shared
00:13:01between different threads and principle you can new table.
00:13:07Bars which are shared between different reads,
00:13:11which is dangerous and this is considered that you should do that
00:13:15at least unless you want really to do things at the lower level,
00:13:19but you can share other things,
00:13:22which are similar to and we are more protected.
00:13:25We will be talking about the art.
00:13:29Of types that we are using the examples vector.
00:13:37Expected shape they are in
00:13:40black list the values and they are associated
00:13:44to integer index so
00:13:46this index index index to you are functional language,
00:13:52you can use the name of this.
00:13:55Array as is the name of the vector
00:13:59as a function function from end to.
00:14:06End if you are an index,
00:14:09which is not there you get out bound and.
00:14:14You can modify vector by
00:14:19changing the association between an index and
00:14:24the values in the previous vector and changing the association
00:14:29to index to say that now becomes the KEE the keyword new name.
00:14:35only to keep it mind is that
00:14:38the language is functional and data structures
00:14:42typically almost all data structures are
00:14:45persistent or not really modified locali the value,
00:14:49but you are at least conceptual qui a new.
00:14:54Vector so if you do this a social and then
00:14:59you look at the same ok under
00:15:05the Hood something it is not your creating
00:15:07a completely new completely new black some sharing
00:15:13and the persistent vector are
00:15:15employment which sharing some some of the tree,
00:15:18but ok from the point of view of the programmer.
00:15:20You can that this is a completely new value.
00:15:26And then we also use maps.
00:15:31Mapping keys to values this is
00:15:36the syntax of maps can be used as functions from keys to values.
00:15:42In this case. If you refer expected if you refer a KEE,
00:15:47which is not there got near result.
00:15:51This is not an error.
00:15:52If you want to get something something
00:15:55different default value when the KEE is not
00:15:57defined in the map you can specific with the sort of coma of steel.
00:16:07Functions in functional language.
00:16:11You want to define functions
00:16:12named functions are defined by the sort of
00:16:15lambda expression syntax is
00:16:20the keyword and the last of parameters possible function is doing,
00:16:26which is then accessible through
00:16:28the doc command And
00:16:31then a list of expressions in this case, we have only one,
00:16:34but it generally list of expressions,
00:16:36which are sequential and value of the function is the value of
00:16:41the last expression and this is taking the.
00:16:47I don't know. How to read the program like this is on your that
00:16:58you can read from inside this is
00:17:00one hundred and multipli by you can read it from
00:17:05the exterior public company is the best program
00:17:09should this expression should be to becomes it difficult.
00:17:15Functions are you can associated to arts and it
00:17:21is a common to the function and given
00:17:24name that you have a shortcut for that the.
00:17:29Io. Functions you can't
00:17:33give different definition for different numbers of parameters.
00:17:39Maybe we just trust that the language is
00:17:43not automatic functions or not automatic carry.
00:17:47IT is not what if you have functional to
00:17:49parameters and only one you are
00:17:52applied function function parameters you
00:17:56can apply the function but to do it explicit.
00:18:05And then functional language.
00:18:08We want to do recursion.
00:18:10We will be using control mechanism,
00:18:14which is mostly sugar and we don't
00:18:17comment on the expected meaning if
00:18:21as boolean expression expressions
00:18:26and you take one of the other
00:18:27pending on the outcome of the boolean.
00:18:32Here we are the some of the first integer.
00:18:38If axis positive then this is the sound of the
00:18:45same of integer up to each other wise it is good.
00:18:53Fibonacci numbers for recursion you can't avoid to about
00:18:58Fibonacci numbers so the Fibonacci over and
00:19:04is the pending on weather is one and is
00:19:08one to one otherwise it is better equal
00:19:13to the some of the two Fibonacci Fibonacci number
00:19:18one and minus. With this.
00:19:35Poi Fibonacci number of time.
00:19:40Of twenty. Thirty walking fine.
00:19:48Fifty. Taking a bit long.
00:19:57Because recursion is very beautiful.
00:20:03But here we are.
00:20:06Computing this is very natural very high level,
00:20:09but per computing you know this computing a lot of
00:20:13time is the same Fibonacci number this is not
00:20:18the implementation of Fibonacci and also this is not so with
00:20:23the right to face this problem
00:20:26and the fine a version of the function.
00:20:30And rather than the first version with the emulator and then he is
00:20:38the functions by giving the right to the way
00:20:42to find the function as an helper function.
00:20:48So Fibonacci as the value KEE start up to zero to accumulate
00:20:58with the Fibonacci, number of KEE and
00:21:03the number and next is the next Fibonacci number one minus,
00:21:09che so the beginning when KEE is this will be
00:21:13the Fibonacci number one and then we start
00:21:18if zero then currencies Fibonacci
00:21:22over and this is the current otherwise we iterate.
00:21:28The next becomes the current and next number is the name of the
00:21:35current plus the next key and the mansion before the beginning.
00:21:40We feed over the two simulator are current next.
00:21:46Zero one anche starts from the details are not really important.
00:21:55This is now in the shape. Here.
00:22:21È un execute fifty fifty much better.
00:22:30Over. One thousand over the thousand.
00:22:41Thousand stack overflow which is a bit display
00:22:47because we designed over functional in the shape.
00:22:54And but if you look at the combination of
00:22:58closure you see that there is not call optimization.
00:23:04You remember this is that when
00:23:09the last call is in the position rather than heading a frame on top
00:23:14of the stack for the new call you replace
00:23:17di back up the color with the state of the function has
00:23:20been because the return value of the college of
00:23:24the school is the return of the college of this,
00:23:28but actually functions translated into methods at the level
00:23:34of the JVM and is not providing tail call optimization.
00:23:40To save the functional language.
00:23:46Provided a way of overcome the problem by employment
00:23:53yourself the optimization as a loop with loop records.
00:24:00So this is mostly automatic.
00:24:04Maybe it is most automatic the position which would have
00:24:08the function definition insert loop
00:24:12with parameters and their initial value.
00:24:17And in the position where you would have
00:24:19the record call insert recall with
00:24:21the new value for the parameters essentials.
00:24:24You are translated yourself the function into loop.
00:24:30This is almost automatic on the wonder,
00:24:34while this is not done automatic.
00:24:38I cannot give you answer, but ok as.
00:24:45Having sample processing which
00:24:49from the cold in this form would be possible,
00:24:53but the only for self calls functions which are.
00:25:01Functions calling themselves,
00:25:05but it would be difficult to employment in full generalita
00:25:08while typically call optimization is
00:25:12it to work as last call optimization so whatever even.
00:25:19So wanted either to provide full time call optimization or not
00:25:27to avoid to give point impressions to programmer
00:25:30decided to give no if you.
00:25:35Want to get better yourself understand.
00:25:44Now this is working better life is record.
00:25:53You gatta every large number.
00:26:08Of features what we just comment marginali
00:26:12because we want be using very much and that
00:26:14this is what we already before code and that are the same
00:26:19least is a program and the last but not the.
00:26:29Least read it is a program where the first element is
00:26:33a function and other are the arguments.
00:26:36The function so if you want to process
00:26:38this least peace and its evaluation.
00:26:42We have to it with the apostrofi
00:26:46or the keyword what not the least evaluate to
00:26:50the evaluation is the principle you can
00:26:55manipulation to some kind of
00:26:57transformation and finally execute the could be the wall.
00:27:02And you can image what you can do a lot of
00:27:07nice and also bad things with this
00:27:10for you may take program which is taking some.
00:27:16Of two numbers, which are constant in this case in general program,
00:27:22which is playing a function and you want to
00:27:25do some kind of optimization could.
00:27:29You replace the function with the optimal one in
00:27:33the world just replace the place with
00:27:36the times and finally you execute
00:27:40the code so the language is really highly reflections.
00:27:46And something which we want us in
00:27:50our example is the take that you can fully use the.
00:27:56Old Java library, which are the level of
00:28:01the JVM you can at the level
00:28:04of the language create objects with Java objects.
00:28:07This is possible syntax,
00:28:09we are creating here you can use idiomatici syntax,
00:28:15which is the name of the class as constructor,
00:28:18which is the end you can associate a bar and then access to the to
00:28:27the object that you created by using
00:28:29the middle class only
00:28:33being careful that this staff is not longer persistent.
00:28:37This is an standard Java is modified by side effects.
00:28:44What we want by doing This is that
00:28:49we the rest of the lesson by
00:28:52talking about the first of the two items that we had.
00:28:58In the plain which is understanding how.
00:29:03The functional can be nice combined
00:29:07with concorrenza is on that you can have
00:29:11automatic parallelizzazione of your code
00:29:15and this kind of released by
00:29:17the infrastructure of closure
00:29:21and we will talk also about the language is not lazy,
00:29:25but sometimes want to have
00:29:26lazy behaviour and we have some support for.
00:29:33The first main point will be the idea
00:29:37that we are mention over and over the
00:29:41same in functional language you copywriter
00:29:43the modified table no side effects you can think having.
00:29:50Current versions of construction,
00:29:54which are typically they are in functional language in
00:29:58particular reading mention If you want to
00:30:01apply function and function function.
00:30:06element collection you see
00:30:09no reason why this should be sequential and.
00:30:13IT is very natural to make it current
00:30:16you think the same for reduced in other languages could
00:30:21fold year old as different name will be
00:30:24exploring habit this time playing with the language by.
00:30:30Understanding how is it is to get
00:30:33the current version of your code when you design
00:30:37your your program in a suitable way
00:30:41and for this is what is the language is not lazy,
00:30:45sometimes if you have a expression rating
00:30:49a very large collection and you want to use
00:30:51only two elements of one to one million.
00:30:53Maybe it's not a good idea to fully
00:30:56it and also if it is not a good idea,
00:31:01if the collection is infinite.
00:31:05And so closure.
00:31:07Which is in the language,
00:31:08which is this is kind of their by
00:31:11design the possibility of elements on demand.
00:31:14Instead closure which is not lazy as an lazy interface
00:31:17to most composite types,
00:31:22which is the lazy sequence interface.
00:31:26But we will talk about the first point and
00:31:29we are doing essential to exercise
00:31:32is which mixing one with the other of this want to computing.
00:31:40And so that with the simple exercise,
00:31:44which is taking the some
00:31:46of the sequence of a collection of numbers.
00:31:50This is the most immediate implementation,
00:31:54which is the collection of numbers empty is the other wise.
00:31:59IT is the some of the first of the numbers
00:32:02and the overall some of the rest of the collection,
00:32:07but this is not so make it.
00:32:11There is what is before and function,
00:32:16which is an accumulatore additional parameter accumulatore
00:32:21which keeps the some of the elements that we explorer up to
00:32:25now and the rest of the least and the rest of the list
00:32:28is that the simulator is the same overall some
00:32:33of the least otherwise there is where
00:32:37the simulator is updated
00:32:40by the first element of the least and that part of the list,
00:32:44which remains to the Explorer is the time.
00:32:50This function the some of the list of numbers is the where
00:32:56the world list Explorer is the list of numbers and the is in.
00:33:05This idea of having a collection of elements that you want
00:33:14to explore accumulate the effect of
00:33:18elements over a single other elements example,
00:33:22which is very commons common typically.
00:33:27Mention that the function is not really there is.
00:33:34What I want to say is that this idea of the least and accumulate
00:33:40the compression say the elements
00:33:44of the least over the single elements,
00:33:47which merge the characteristic
00:33:50of the elements of the least is something
00:33:52very common so common typically functional language order
00:33:57construct which Doing exactly the thing,
00:34:01which is called reduce other languages it for the year old as
00:34:05a different meaning reduce taking a function and
00:34:11the list of elements in initial value for the other and
00:34:15then the function is used for accumulate in a single value.
00:34:21See the effect of the elements at least so apply to it and value.
00:34:29Then again and the second argument and
00:34:36so so you reduce the least
00:34:40to a single element is the final value of the simulator,
00:34:45which is exactly what you are doing before where was the same.
00:34:51And elements where the numbers
00:34:55that we want to take the sound of so we
00:34:59can use rather than doing recursion then about.
00:35:08The call optimization and moving to the record loop version.
00:35:14We can just use reduce we can reduce
00:35:17the list of numbers by using reduced function
00:35:20the same starting value zero list of
00:35:24numbers and what we are
00:35:27doing is exactly what you want from the wall
00:35:29of labor at the end to end to end
00:35:36so up to the last element so your accumulate the some of numbers.
00:35:46And you could avoid to this is
00:35:51just a minor observations that some Built in sum in svariati
00:35:57cambi Binary so you don't need to find
00:36:01yourself you can you want you can also
00:36:07be execute without specific initial value in this case
00:36:11the same in particular list is
00:36:16contain at least one element this starting
00:36:20rather than from initial value from the first element of the
00:36:23least so you can avoid
00:36:25specific initial value so this is very very contact, but.
00:36:34IT is not current,
00:36:37but how can I reduce concorrenza this
00:36:40looks really an intrinseca sequential.
00:36:44Operations which is true,
00:36:50but the function as some properties for
00:36:53instance associative you think of splitting the least in
00:37:00bounces in chunk and reduced
00:37:03the single chunk currently and everything together
00:37:08may be again can country it really
00:37:12not and so I'm principal there is for making this current,
00:37:17but for now, we didn't and the other level contract,
00:37:23which is typical or functional languages is not least
00:37:28from playing single function to all elements again,
00:37:35not very natural for making it current.
00:37:43But for the moment it is not small example
00:37:47here image combined reduced map.
00:37:52imagine you want to computer model of the vector.
00:37:56This is not a great amount of work,
00:38:00but steel we can as it is an exercise to do it
00:38:05with the tools to know
00:38:07how you want to take the power of the same square.
00:38:11So what you do you need a functions for square in numbers,
00:38:15Then you map dysfunction over the components.
00:38:20Thank you reduce the result with the
00:38:23same so you get some of the square and that the spirit.
00:38:28Of everything is still sequential might think.
00:38:35We want to make the current how may be rather than playing
00:38:43if sequential first to one to one
00:38:46two four and so on program which is this is.
00:38:59What we will understand what we
00:39:02don't need to do it yourself because it is already there,
00:39:05but before understanding we take more complex example allows
00:39:12to to make also some experiment with
00:39:16timing and classical example in the setting,
00:39:21which is computing frequency of
00:39:24large text so you want to know how frequently a world
00:39:30in given text so given large text to be large you want
00:39:39to identifying the number of currencies of
00:39:42lead single world in the text for instance
00:39:44from the cat is on the table you want to get the map.
00:39:48where the KEE is the world's
00:39:51and the values associated the keys are the.
00:39:56Number of currencies of the world in the text.
00:40:03For the use of nations on maps.
00:40:07You remember this is the way of the maps.
00:40:10We will use the syntax for access rather than the functional one
00:40:14for the values KEE where you can
00:40:18also specifi a default value in case that the KEE is not associated
00:40:24anything map so for you will for which is not in the map you.
00:40:33And remember what we can modify the map,
00:40:36but this is not really local modification,
00:40:39you can generate new map where you
00:40:41change the association with the KEE.
00:40:43This is that we will be using.
00:40:47So we can program step by step.
00:40:53AM. In order to get.
00:40:59The frequency.
00:41:02Of in the text,
00:41:06which is already splitting words so this is not string,
00:41:10but sequence collections words We can use reduced.
00:41:17Where you want to use this collection words in the map.
00:41:23For reducer function use function which given
00:41:27a board and current map updated map
00:41:30by using one to the corrispondente.
00:41:34The definition is not super interesting so
00:41:37maybe we just keep it or maybe it.
00:41:41This is this function Given a world you want
00:41:46to update count up by update.
00:41:49The KEE world of it is ready or it to one it.
00:41:56IT is not so you access
00:42:01map in the point to the keyword and the corrispondente value zero.
00:42:07If it is not there is incremento because you
00:42:10have a new world of the mind and finally you get
00:42:13the new world has been
00:42:16updated the number of the world has been updated in this way.
00:42:24World is to a sequence of world what you want.
00:42:30The frequency of world in the map,
00:42:34but we want to string to text and so we want to
00:42:40the first transform that text into collection of
00:42:43world and then the world is the world is not building,
00:42:48but you can build it yourself.
00:42:52Now actually what we want to it something more difficult which
00:42:57is counting world for text of several strings.
00:43:04So what we can do is given the sequence strings map worlds to
00:43:10the sequence strings widget sequence of sequence of words.
00:43:16That we map world frequency.
00:43:20And we get sequence of maps each
00:43:24providing the frequency of each string and we
00:43:29want to merge the corrisponde
00:43:33in maps how taking the Union of the keys.
00:43:38And for common you want to take
00:43:40the And for this you have a built function almost which is it.
00:43:49IT takes a list of maps.
00:43:51The second maps collection of maps and the function,
00:43:54which is used for
00:43:55combined keys values for common keys so you start from the first to
00:44:02map it together for common
00:44:06you apply to the corrispondente values
00:44:09to them together in our case.
00:44:11This will be the same.
00:44:18And so what we need is a partial instance of this
00:44:23is where the first argument is some that the language is not
00:44:28qualified functions are not so you need to apply
00:44:32explicit partial way the function partial
00:44:36apply partial image to plus
00:44:38so you get the Function is
00:44:42only maps use plus for common keys and we can eat.
00:44:51This is to the following maps keys.
00:44:58For common keys you get the same.
00:45:03And we are.
00:45:08Power. Because for
00:45:16computing the frequency in sequence streams.
00:45:20We do what we said we map get
00:45:24the sequence sequence of boards that we
00:45:27map frequency getting a sequence of maps,
00:45:31the frequency of x single string and finally we use as.
00:45:38the reducer much count and
00:45:41reduce the least the sequence of maps to single not.
00:45:50Back up to know a starting talking about there is not here.
00:46:00To make it.
00:46:06Out one we have a sequence of strings,
00:46:12which is in the only of two strings and what we are doing.
00:46:17We are playing networks to the first time to the
00:46:21second to the world's now.
00:46:27We go back to the first sequence of
00:46:30the world frequency to the second worlds world frequency
00:46:34and so on Finally we put together or
00:46:38this maps and we have only two,
00:46:41but in principal you would have
00:46:43many and also this merging might be perform some how.
00:46:49Maybe first merging several splitting
00:46:54the different maps in churches
00:46:56merging the first chunk the second chunk
00:46:59the chunk currently thank you get the list of maps, which could.
00:47:06And this looks natural and that is natural,
00:47:10which is kind of almost built in the language.
00:47:16Mapping in the way to
00:47:20like the great idea what we have it is not our idea.
00:47:25IT is ready, there is the language,
00:47:28we have parallel but rather than playing
00:47:31the function first to the first time to the second element and
00:47:35the element rather created a number
00:47:38of processes actually and the good something this are not
00:47:43system threads del pool of workers to meet
00:47:46the jobs or summit in principal current process.
00:47:53Taking of applied function to
00:47:57one element of the least and the results are connected.
00:48:02If you do it ok in this case amount of work is not
00:48:06very meaningful to be to use parallel map,
00:48:12but just to simulate the fuck you want to
00:48:16do work with rather than incremento we use a slow incremento,
00:48:22which sleep bit one thousand seconds,
00:48:25one two seconds and then from the actual inch to this.
00:48:42And then we want.
00:48:47To what we need some elements we range
00:48:53which producer sequence number start up to one specific.
00:49:03So we want to this.
00:49:11And its first man in the first and the second it will
00:49:15take the seconds that we used parallel version.
00:49:23IT is taking proximity
00:49:26one second growth so we are super happy about this.
00:49:36Pink of heaven and the
00:49:39problems rather than using
00:49:42the sequential version with defined before
00:49:45we USA version just replace map it
00:49:49up. So super simple.
00:50:06And then. Here we have a cut.
00:50:15Bar which is strings which contains one thousand strings of the
00:50:21world's world's consistent with
00:50:26means you will have the number of reputation.
00:50:30This is the string is not really very interesting
00:50:36and we use it for our benchmark and so we start to the world's.
00:50:49Sequential over strings.
00:50:54Indicator, but it is not to see you have a lot of.
00:51:01Things are generally random there is what
00:51:06we want to do some timing here and we don't want to see the output.
00:51:10We will be our.
00:51:15Timing we associates and result with the bar.
00:51:25And this is taking almost two second
00:51:28when we are very excited because we have the parallel version.
00:51:39That is really decrease in very fast
00:51:43because this is not much more efficient.
00:51:50Get back and we look at our function and we have a first intuition,
00:52:01which is we're going to
00:52:04the sequence to the collection of strings and playing in
00:52:08parallelo to the strings get the result you get a collection
00:52:15of sequence of sequence of
00:52:18words and you get through
00:52:19the collection and apply in parallel frequency.
00:52:22Maybe this is not the best you can do First point might be
00:52:29not the composition of
00:52:32the world's world frequency to
00:52:34each collection to each string this might.
00:52:39And we do it so this is
00:52:43the world's parallel component together the functions, we know how.
00:52:50It's not really important.
00:53:03Better it better, but not we have eight course making things
00:53:10current can a little bit more is not anyway one of more.
00:53:22But it Point you have to remember that
00:53:26the strings are of twenty words,
00:53:30which is really little works to be
00:53:33done and we are creating a lot of process.
00:53:37Whatever they are even if they are jobs,
00:53:40but still you need some coordination for this
00:53:45may be creating one process
00:53:48for this is too much is not the good idea.
00:53:52So try to give more work to each single worker.
00:53:59So possibility can be to take
00:54:05our input sequence of strings splitting
00:54:09chunk in batch size and and
00:54:14process it chunk by using sequential sequential version,
00:54:22But the different sequential version applied to
00:54:25the church this myth some improvement.
00:54:31About the way splitting sequence in common of this is not.
00:54:46So this is it called.
00:54:51Butch. Where to specifi
00:55:02how large are the string it is
00:55:06one such a great ideas because we are doing
00:55:12the same things before creating ten thousand process now is
00:55:18more work to work so
00:55:21ten strings per worker which
00:55:24means we are creating ten thousand process.
00:55:28It's getting better one ended means
00:55:32one thousand process it taking some amount work one strings.
00:55:41And processes sorry one thousand size for
00:55:45the process getting even better.
00:55:52Than thousand much better.
00:55:58Than which means that we are creating the process which
00:56:03processing ten thousand strings fully using our.
00:56:11Course if we go higher
00:56:15twenty thousand that we are Splitting Thanks five process,
00:56:21we expected increase in performance because we are
00:56:24not fully using the course.
00:56:28Five fifty thousand nice to thanks to using only to the world's
00:56:36and we go to one and the thousand which means
00:56:40a single chunk because this is the strings that.
00:56:47We are back to the sequential version and this is
00:56:53the time actually this little wars because there is
00:56:55also some additional infrastructure.
00:57:06And how maybe a final four about this is.
00:57:11The kind of cooperation,
00:57:14which is what we are doing,
00:57:15which is Reduce a large amount of data two single values,
00:57:24which with the number of which is to be
00:57:28together is not something so unusual so.
00:57:35Actually it happens often what you
00:57:38have a large collection of things and what you want to do
00:57:43is to split this collection of things and
00:57:45now the strings into the church and you want to
00:57:53reduce is back by using a given function
00:57:56g. This provides a number of values of some kind
00:58:03that you want to combine together so this is a sort of
00:58:08synthesis of chunk and then you want to put together
00:58:11to merge on the synthesis coming from the different,
00:58:14thanks for this you want to use a different function
00:58:16typically not necessary, but sometimes.
00:58:23And this is called fault this again,
00:58:27very natural and food and and setting of.
00:58:36You can you need for the original problem.
00:58:44Food is using the same function for you take large list of
00:58:52numbers hit list of number is reduced
00:58:55by taking by using place to get some of the chunk.
00:58:59So this is what you want to combine together and you use again.
00:59:03Saint just insert single function it is for education and merge.
00:59:11And what is is that everything is in the best possible way.
00:59:18And so if you compare but before by using the.
00:59:25The manual implementation.
00:59:28This is what the time is what you get with
00:59:30full implementation by doing nothing apart from the problem.
00:59:35A suitable way you read more less by the number, of course.
00:59:41Can we do the same also for the original problem frequency.
00:59:47Yes, we can in different ways one possible way is to.
00:59:52The sequence of strings and construct
00:59:56single sequence of
00:59:58world's first splitting words and different sequence.
01:00:06And then let fold to the rest of the job.
01:00:11by out different church of
01:00:15these worlds reduced by using one of the functions with defined
01:00:20before and counting the one we
01:00:23were dating over and over by the king care of
01:00:27the different currencies and then we
01:00:30use merge with by the different maps.
01:00:35So this is.
01:00:38The overall program actually forget about this is just functions
01:00:45defined before counting count and we do is the sequence of things.
01:00:51We map it to map that is doing map it first mapping function
01:00:59to this getting sequence sequence merging
01:01:02Barakat The sequence is in
01:01:04the single sequence of world of the year,
01:01:07you have a sequence of world of the strings and you just to fold
01:01:12by using counting as reduce and merge count merger.
01:01:25And very.
01:01:28IT is very satisfaction that actually this is given the best timing
01:01:35without almost any effort apart from structure
01:01:38of the code in the way.
01:01:55Of this is more it will we discuss just about the last point,
01:02:01we want to discuss today lines.
01:02:06Is not lazy it is not if you define say a function,
01:02:15which is the constant zero.
01:02:17This doesn't look at the argument in principle
01:02:20that if zero to
01:02:23whatever expression expression is not evaluate if the language is.
01:02:28This is not because this is defined function.
01:02:38And then apply to
01:02:40this function we don't get zero rather stack overflow.
01:02:47Which means that this is evaluate.
01:02:51But as we mansion before.
01:02:54The view of collections
01:02:59is not really typical sometimes not the best Typekit you want to do
01:03:04because you Haven expression which is
01:03:07very large collection and you want to take the three elements than
01:03:12the food collection and only three remains might not be
01:03:16the good idea and even works when the collection is
01:03:21infinite and so actually even if up to now.
01:03:30We didn't make this explicit several of the functions that we
01:03:34know are not really lists or array or map,
01:03:40but rather their produce lazy sequence is you have this interface,
01:03:47which is an interface towards several types.
01:03:51The language arrays vector lists maps,
01:03:56which view this structure in ways
01:04:00dei keep the expression And only generated elements
01:04:06on demand so we can understand
01:04:11this examples if you define we have range is one under.
01:04:20This is the least we define numbers to be.
01:04:26Elements consider the times by the takes to
01:04:30generate almost nothing and one thousand elements.
01:04:38Brilli ten thousand elements very little
01:04:43time one and the thousand one million elements.
01:04:48This is more and more and the point that
01:04:51actually this is not really rating and the sequence,
01:04:58but it is better it is not a Data structure
01:05:02but if you look at documentation,
01:05:07this is and that it is the sequence.
01:05:10What is that you are just remember the code,
01:05:16but it associated it to
01:05:18the thing is not evaluation and it will be evaluate on demand.
01:05:25If you know how access to the same element that list have
01:05:31one billion access the second element of the list of numbers.
01:05:40As we will be probably
01:05:44fast because it is the beginning what if you access to the element.
01:05:54Close to the end of this sequence
01:05:57that you have to generate all the sequence.
01:05:59Ehm. If. You want the sequence to be fully real
01:06:07as you can you do this
01:06:11is the sequence and not no longer in way,
01:06:18but it is completely this is the time it
01:06:22takes and so sequence is are lazy a vector are not to take your.
01:06:33Sequence numbers. And sei.
01:06:41Put your sequence into an array.
01:06:48So numbers are this? Array No.
01:07:00This is taken sometimes I measure.
01:07:06Some time what if you.
01:07:12Access to the lost element.
01:07:16This is what is access and the second.
01:07:22Ok. What?
01:07:32Is Okay having staff is
01:07:37convenient staff is large and you want to use if you want to.
01:07:45Take the some of the food
01:07:50and unless this is huge so what you can be in
01:07:57Full generated does not really make
01:08:01much science because you will have to generated but
01:08:06ok if instead sequence is that it is better not fully
01:08:10generated as you can imaging and support for this sequence.
01:08:16Please sequence can be infinite and you have.
01:08:20Comments which are intended to support
01:08:23the creation of infinite sequence
01:08:26is for instance hit rating function over the given initial value.
01:08:31This is the function
01:08:34of the function against the results and so on support.
01:08:40And clear you should never fully sequence only take
01:08:44some element or you
01:08:47can apply function with no arguments forever repeat.
01:08:53This may This is a stream of random numbers,
01:08:58which again you should never access
01:09:02entire but only one random number after the other.
01:09:09This might be able to you, but anyway,
01:09:12Let's comment about it
01:09:13quickly let's go to transform such a sequence our numbers,
01:09:19which is one million numbers and we want to incremento all of them.
01:09:28Because this is the shift number. Of the time.
01:09:39And again super fast.
01:09:43While instead we do the same for.
01:09:49Grey and we use map because otherwise also map it.
01:10:01Takes the time, but it is needed
01:10:05and this might be completely to you the point is that.
01:10:11AM. On sequence in
01:10:17the same way as you want to the sequence has just recording
01:10:22the generator sequence without
01:10:24the elements transform sequence is just apply transformer to
01:10:30the generator and this is recorded
01:10:34never really unless you start access and
01:10:38the elements of the intuition and the complex,
01:10:47but the intuition is that sequence is to
01:10:51Das generator of the sequence and the transformer reducer and then.
01:11:00When you transform the sequence for
01:11:05this mapping functions sequence you're
01:11:08just computing the transformer of
01:11:11the sequence with the function mapping this takes
01:11:15essential time takes time only when you
01:11:18explicit refit by access to the elements.
01:11:25I think that this is more less it in
01:11:28case you want to know more it is
01:11:31left to you and the one other possible designing for the.
01:11:39Getting more in detail into
01:11:42some of the languages that we explore some of
01:11:45those that we will not for today think this is enough.