First of all, in such a situation you will have to pass the “return” parameters “according to a pointer”. Since the "returned" parameters themselves are of type int ** , then by passing "by pointer" they will turn into int ***
Secondly, in order not to be confused with asterisks, I would recommend making memory through the following idiom
dst = malloc(N * sizeof *dst);
(and at the same time get rid of the manners to explicitly give the result of malloc )
void ArrayInput2(int ***pX, int n, int m, int ***pY, int a, int b) { ... *pX = malloc(n * sizeof **pX); for (int i = 0; i < n; ++i) { (*pX)[i] = malloc(m * sizeof *(*pX)[i]); for (int j = 0; j < m; ++j) fscanf(file, "%d", &(*pX)[i][j]); } ...
Third, as an optional modification, you can offer to allocate second-level memory as a single unit, rather than with multiple calls to malloc
void ArrayInput2(int ***pX, int n, int m, int ***pY, int a, int b) { ... *pX = malloc(n * sizeof **pX); **pX = malloc(n * m * sizeof ***pX); for (int i = 0; i < n; ++i) { (*pX)[i] = **pX + i * m; for (int j = 0; j < m; ++j) fscanf(file, "%d", &(*pX)[i][j]); } ...
When calling such a function, of course, one must not forget to pass exactly pointers to pointers-receivers.
int **X, **Y; ArrayInput2(&X, 10, 20, &Y, 30, 40);
But, generally speaking, the code would have to be redone differently. A function that reads two matrices should not be. There must be a function that reads only one matrix from the given file and returns it via the return value.
int **matrix_input(FILE *f, int n, int m)
Then the "stars" of these high-rise would be less.
int **. - Igor