A polyphase sequence set with orthogonality consisting complex elements with unit magnitude, can be expressed by a unitary matrix corresponding to the complex Hadamard matrix or the discrete Fourier transform (DFT) matrix, whose rows are orthogonal to each other. Its matched filter bank (MFB), which can simultaneously output the correlation between a received symbol and any sequence in the set, is effective for constructing communication systems flexibly. This paper discusses the compact design of the MFB of a polyphase sequence set, which can be applied to any sequence set generated by the given logic function. It is primarily focused on a ZCZ code with q-phase or more elements expressed as A(N=q^n+s, M=q^n-1, Zcz=q^s(q-1)), where q, N, M and Zcz respectively denote, a positive integer, sequence period, family size, and a zero correlation zone, since the compact design of the MFB becomes difficult when Zcz is large. It is shown that the given logic function on the ring of integers modulo q generating the ZCZ code gives the matrix representation of the MFB that M-dimensional output vector can be represented by the product of the unitary matrix of order M and an M-dimensional input vector whose elements are written as the sum of elements of an N-dimensional input vector. Since the unitary matrix (complex Hadamard matrix) can be factorized into n-1 unitary matrices of order M with qM nonzero elements corresponding to fast unitary transform, a compact MFB with a minimum number of circuit elements can be designed. Its hardware complexity is reduced from O(MN) to O(qM log _q M+N).