. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . COMP6700/2140 Correctness and Efficiency Alexei B Khorev and Josh Milthorpe Research School of Computer Science, ANU May 2017 Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 1 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Topics 1 Concept of an Algorithm 2 Correctness 3 Cost of Computation: Time, T Space (Memory, P) Energy 4 Efficiency and Complexity of Algorithms 5 O-expression Notations 6 Complexity Classes 7 Linear Search and Selection Sort 8 Performance Measurement 9 Bubble Sort and Gap Sort Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 2 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithms: Top-Down Design (Again) Algorithms occupy the intermediate position between a solution and a method, both in terms of generality and detail. The origin — mathematical (algebraic), the essence — transformation of data (input → output) in a finite number of steps. Writing a computer code often involves solving algorithmic problems. Some of these problems can be handled by the top-down design approach: 1 Clearly state the problem that you are trying to solve 2 Define the inputs required by the program and the outputs to be produced 3 Decompose the program into classes and their associated methods 4 Design the algorithm that you intend to implement for each method Repeat every step for devising the solution used at the higher level (step-wise refinement) 5 Turn the algorithm into pseudo-code (easily translatable into Java statements) 6 Test the resulting program The top-down design is often used together with the bottom-up design — translating an existing solution (like a mathematical solution into the code and integration separate parts into one working code). The bottom-up design may also involve the data structure selection (or construction); this is called the data-driven design. Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 3 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithms: generality A good algorithm always has a virtue of generality: it can be used on a family of data structures with specific characteristics reduced to minimum. An example — sorting an array of N floats: This problem is a particular case of a more general sorting problem: Given an array of N > 0 objects, return an array which contains these N objects in ascending (descending) order. It is assumed that array elements can be compared, using unspecified criterion (in Java, it means that the object class implements the Comparable interface). When implemented with such generality, the algorithm is said to be generic. Computer algorithms are characterised by: 1 a large number of repetitive operations: large size array, looping constructs, pipelines 2 a particular data model in case of sorting and searching (“data model” here means type of operations which can be performed of the DS) 3 correctness and efficiency, where correctness means that a right result is obtained for all possible problem instances (size of the data structure, state of the input — sorted, unsorted, random, nearly-sorted etc), while efficiency means how the time and computer resources needed to successfully carry out the algorithm application scale with the problem size N (number of elements in the DS). Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 4 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithms: Complexity To quantify the algorithm efficiency, one has to establish how T(N) — the number atomic operations (they do not depend on N), needed to perform the algorithm — scales with the size of the problem N. The time of execution depends on hardware platform and state of the execution environment; it is not an invariant measure. The computer science uses the big-O notations (pronounced “big-omicron” by eggheads, “big-o” by pub frequenters) to characterise the efficiency (complexity) of an algorithm. It tells what is the leading term of the scaling dependency: if lim N!1 T(N) = const f (N); thenT(N) = O (f (N)); when N becomes large enough. The dependence T(N) of an algorithm, viewed as a problem, is referred to as complexity viewed as a solution, is referred as efficiency Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 5 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Complexity classes The function f(N) usually has one of the following types of behaviour: the power law: f (N) = N ; where is a number equals to 2 for the Selection Sort, and to 2.376 for “smart” matrix multiplication, etc. Such polynomial problems are tractable. the logarithm law: f (N) = (log N) ; where is 1 for a binary search (in a sorted array, in a binary search tree). These are easy problems. the exponential law: f (N) = 10 N; where is 1/3 for the factorisation problem. Such problems with exponential behaviour are hard (intractable). Examples: the traveling salesman problem, set partitioning etc. A combination (as factors) of the above, eg, f (N) = N log2 N; for the QuickSort algorithm f (N) = N3(log N)k; for the quantum Shor algorithm Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 6 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How complexity matters Logarithmic scale in the ordinate axis! Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 7 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ο, Ω and Θ Some rigour To proper understand the O–notations one should remember that O(f(N)) denotes not a single function, but a class of functions with the designated asymptotic behaviour. If we say that a function g(N) has an asymptotic of O(f(N)), g(N) = O(f(N)), it means not equality, but rather the inclusion of g(N) into the same class. The g(n) = O(f(N)) has meaning of the upper-bound, ie that its magnitude is at most as big as const×f(N) (when N is large). At most! but it could also be (substantially) smaller. Hence, an algorithm should not be dismissed outright if it is characterised by the efficiency of, say, O(N2) (like in sorting), just because its actual performance can be better. Opposite to upper-bound is an lower-bound estimate, ie that the function magnitude is at least as large as const×f(N) (N is large). This asymptotic behaviour (different set) is notated with another symbol, Omega: g(N) = (f (N)). The -properties of algorithms are harder to establish, but they are more valuable. Even more valuable (and still harder to come by) are asymptotic with both lower- and upper bounds: that a function g(N) is such that for sufficiently large N, there exist two constants, C1 and C2 (C1 > C2), that C2 f(N) < g(N) < C2 F(N). Such asymptotic sets are notated with the symbol : g(N) = (f(N)). Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 8 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . “Greek” Asymptotes bounded from above g(n) = O(f(n)) bounded from below g(n) = (f(n)) bounded in between g(n) = (f(n)) One can proof that (something which is intuitively obvious): g(n) = O(f(n)) and g(n) = (f(n)) () g(n) = (f(n)) The reason (not because “programmers are stupid”) of why the O-notations are used predominantly (when would be more accurate) is explained in a classic Donald Knuth’s paper “Big Omicron and Big Omega and Big Theta”, in SIGACT NEWS, Apr.-June 1967, pp.18–24. Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 9 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linear Search Problem: Given unsorted array of N elements (double’s), find the smallest element in a subarray. Algorithm: (trivial, correctness follows from the facts that each element of the array will undergo a comparison with the tentative result and comparisons of integers and real numbers are transitive, ie from a> b and b > c follows a > c). 1 Assume tentatively that the first number in the array is the smallest element 2 Compare this number with every element in succession If one of these subsequent numbers is smaller, replace the tentative choice by this element 3 When all of the index locations have been exhausted, return the tentative result /** @param data an array of doubles * @return value of smallest element */ public static double smallest(double[] data, int lo, int hi) { assert (hi > lo); int iSmallest = lo; //Initial value for (int i=lo+1; i<=hi; i++) if (data[i] < data[iSmallest]) iSmallest = i; return data[iSmallest]; } Complexity of this algorithm is O(N): we have to go from the first element to the very last one in search of the smallest (unsorted nature of the array is essential). Alexei B Khorev and Josh Milthorpe (RSCS, ANU) COMP6700/2140 Correctness and Efficiency May 2017 10 / 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SelectionSort 1 In a loop which iterates a counter, i, over every index of an array in ascending order 2 At every step, do the following Compute the smallestInRange (the index of the smallest element) between the present array index, i, and the maximum array index. Call this index iSmallest Interchange the array elements at i and iSmallest /** @param data an array of doubles */ public static void selectionSort(double[] data) for (int i=0; i